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Overview 



Project Graduation Really Achieves Dreams (GRAD) is an ambitious education reform initia- 
tive designed to improve academic achievement, high school graduation rates, and rates of col- 
lege attendance for low-income students. Recognizing that high schools inherit problems that 
have arisen earlier in the education pipeline, Project GRAD intervenes throughout an entire 
“feeder pattern” of elementary and middle schools that send students into each high school. This 
report presents results of MDRC’s multiyear evaluation of the effects of Project GRAD on stu- 
dent progress at three high schools in Houston (the initiative’s original site) and at high schools 
in two other school districts (Columbus, Ohio, and Atlanta, Georgia). A companion report dis- 
cusses findings for Project GRAD elementary schools in four cities. 

Project GRAD schools at all levels build support in the community for school improvement and 
college attendance, implement a classroom management program, provide students with access 
to needed social services, and receive special support from local Project GRAD organizations. 
Project GRAD elementary schools implement specific reading and math curricula, along with 
enhanced professional development for teachers. At the high school level, Project GRAD's 
model assumes that better-prepared students would come from the Project GRAD feeder 
schools, would benefit from special academic counseling and summer academic enrichment in 
high school, and would qualify for a scholarship to attend college, which is the “cornerstone” of 
the Project GRAD reform. 

The key findings of this report are: 

• At Jefferson Davis High School in Houston — the initiative’s flagship school — Project 
GRAD had a statistically significant positive impact on the proportion of students who com- 
pleted a core academic curriculum on time. As Project GRAD expanded into two other Hous- 
ton high schools, these positive effects on students’ academic preparation were not evident. 

• Improvements in graduation rates at the three Project GRAD Houston high schools were 
generally matched by improvements in graduation rates at the comparison schools. 

• The Project GRAD high schools in Columbus and Atlanta in the early years of implemen- 
tation showed improvements in attendance and promotion to tenth grade that appear to 
have outpaced improvements at the comparison schools, although the differences are only 
sometimes statistically significant. 
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Preface 



In the past decade, school districts around the country have sought to improve struggling 
urban high schools, where high dropout rates, poor student achievement, and low rates of gradua- 
tion and college-going remain too prevalent. In a field crowded with reform initiatives, Project 
Graduation Really Achieves Dreams (GRAD) stands out as particularly ambitious, focusing as it 
does on improving conditions for high school students before they even reach high school. 

First implemented in one high school in Houston, Texas, in the early 1990s, Project 
GRAD has evolved into a comprehensive reform model that intervenes throughout an entire 
“feeder pattern” of elementary and middle schools that send students into each Project GRAD 
high school. The developers of Project GRAD understand that high schools inherit struggling 
students, making it essential to improve both elementary and secondary schools in order to in- 
crease the rates of high school graduation, college -going, and college graduation. Project 
GRAD combines curricular reforms in the lower grades and the opportunity to qualify for a col- 
lege scholarship in high school with a classroom management program, access to social ser- 
vices, and efforts to promote parental and community involvement at all grade levels. 

With principal support from the Ford Foundation, MDRC has conducted an evaluation 
of Project GRAD in Houston and several expansion districts. This report focuses on student 
achievement at three high schools in Houston and at one high school each in Columbus, Ohio, 
and Atlanta, Georgia. A companion report offers findings on elementary school student 
achievement in Houston, Atlanta, Columbus, and Newark, New Jersey. 

Taken together, the findings from both reports highlight the challenges that urban 
school districts face in significantly improving the academic performance of high school stu- 
dents. While Project GRAD schools made some significant gains in elementary test scores and 
for students at the flagship high school in Houston, in other areas they did not outpace the com- 
parison schools, many of which were also engaged in local and districtwide reforms. Project 
GRAD USA, the dynamic organization that oversees the initiative, has already begun to refine 
its comprehensive approach to respond to some of the lessons suggested by this unusually rig- 
orous evaluation — namely, the “leaky” nature of many feeder systems, the challenges of bring- 
ing reforms to scale, and the importance of taking action at the classroom level to get academic 
gains. Both the national organization and the local Project GRAD sites deserve credit for sub- 
mitting their model to a rigorous comparative study. 

This evaluation of Project GRAD is only one of several studies that MDRC has recently 
conducted of high school and district reform efforts, including Career Academies, First Things 
First, and Talent Development. For more information about MDRC’s education research and to 
download all MDRC reports, please visit our Web site at www.mdrc.org. 

Gordon Berlin 
President 
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Executive Summary 



Project Graduation Really Achieves Dreams (GRAD) is an ambitious education reform 
initiative designed to improve academic achievement, high school graduation rates, and rates of 
college attendance for low-income students. It is an unusual reform model in that it intervenes 
throughout an entire “feeder pattern” of elementary and middle schools that send students into 
each Project GRAD high school. The initiative recognizes that high schools inherit problems 
that have arisen earlier in the education pipeline, making it essential to improve both elementary 
and secondary schools in order to increase the rates of high school graduation, college-going, 
and college graduation. 

Project GRAD schools at all levels build support in the community for school im- 
provement and college attendance, implement a classroom management program, provide stu- 
dents with access to needed social services, and receive special support from local Project 
GRAD organizations. To help students arrive at middle and high school better prepared aca- 
demically, Project GRAD elementary schools implement specific reading and math curricula, 
along with enhanced professional development for teachers. At the high school level, Project 
GRAD's model assumes that better-prepared students would come from the Project GRAD 
feeder schools, would benefit from special academic counseling and summer academic enrich- 
ment in high school, and would qualify for a scholarship to attend college, which is the “corner- 
stone” of the Project GRAD reform. 

This report describes the effects of Project GRAD on student progress at three high 
schools in Houston (the initiative’ s original site) and at high schools in two other school districts 
(Columbus, Ohio, and Atlanta, Georgia). MDRC — a nonprofit, nonpartisan research organiza- 
tion — conducted a third-party evaluation to determine the effects of Project GRAD by compar- 
ing the changes in student outcomes at Project GRAD schools with changes at similar, non- 
Project GRAD schools in the same districts. (A companion report discusses findings for Project 
GRAD elementary schools.) 1 In general, Project GRAD student outcomes are tracked from the 
implementation of the first components of the model at each site until the 2002-2003 school 
year. The key findings of this report are: 

• At Jefferson Davis High School in Houston, the initiative’s flagship school, 
Project GRAD had a statistically significant positive impact on the pro- 
portion of students who completed a core academic curriculum on time 
— that is, received an average grade of 75 out of 100 in their core courses; 



'Jason C. Snipes, Glee Ivory Holton, and Fred Doolittle, Charting a Path to Graduation: The Effect of Pro- 
ject GRAD on Elementary School Student Outcomes in Four Urban School Districts (New York: MDRC, 2006). 
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earned four credits in English, three in math, two in science, and two in 
social students; and graduated from high school within four years. 

• As Project GRAD expanded into two other Houston high schools, these 
positive effects on students’ academic preparation were not evident. Stu- 
dent outcomes at the newer Project GRAD high schools improved, but 
generally this progress was matched by progress at the comparison high 
schools. 

• Improvements in graduation rates at the three Project GRAD Houston 
high schools were generally matched by improvements in graduation 
rates at the comparison schools. 

• Looking at early indicators of student success, the initial Project GRAD 
high schools in Columbus and Atlanta showed improvements in atten- 
dance and promotion to tenth grade that appear to have outpaced im- 
provements at the comparison schools, although the differences are only 
sometimes statistically significant. 

The remainder of the Executive Summary describes the Project GRAD model and how 
it was implemented in the school districts, explains how the evaluation was conducted, and 
summarizes the study’s findings and explores their implications. 



What Is Project GRAD and How Was It Implemented? 

Project GRAD originated in 1993 at Jefferson Davis High School in Houston, Texas, 
when business leaders sought to increase the rate of college-going among the school’s mostly 
low-income students by offering scholarships to students who qualified. When this offer did not 
generate a significant increase in high school graduation and college enrollment, Project GRAD 
was expanded to include school improvement efforts at the associated middle and elementary 
schools that fed into the high school. 

A complex, multilayered initiative, Project GRAD includes a set of core components for 
all the schools in a feeder pattern as well as components for the schools at each level, as described 
below. It is important to note that these components were phased in over time at each site. 

Components at Project GRAD High Schools 

At the high school level, Project GRAD’s strategy relies on the demand push for change 
from better-prepared students coming from Project GRAD feeder schools and the opportunity 
pull toward success provided by the promise of the Project GRAD scholarship. Unlike its inter- 
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vention in the lower grades, Project GRAD at the high school level does not modify the curric- 
ula in school, provide substantial professional development for teachers, change the way in- 
struction is offered in the classroom, nor directly address skill deficits of entering students. In- 
stead, it offers two particular components that encourage planning for college, as well as three 
components that seek to create an environment that is conducive to learning: 

• Project GRAD college scholarships are provided to students who have a 
cumulative 2.5 grade point average, graduate within a four-year time period, 
complete a recommended college preparatory curriculum, and participate in 
two summer institutes. Scholarship amounts and criteria vary slightly by site 
but usually average $1,000 to $1,500 each year during the four years of col- 
lege. Each Project GRAD high school has a scholarship coordinator who 
provides counseling, tutoring, and college admission preparation. 

• Summer institutes provide an opportunity for qualifying Project GRAD 
students to experience a college campus-based program taught by college 
faculty and to enhance their academic skills. The activities vary by site but 
typically include reading, writing, math, science, enrichment, and remedial 
activities. The institutes usually consist of four to six hours of instruction and 
related activities per day for four to six weeks. 

• Parental and community involvement to engage parents and the commu- 
nity in the work of the schools, build awareness of the opportunity to attend 
college, and support the learning of students. At the high school level, annual 
Walks for Success are conducted, in which principals, teachers, Project 
GRAD staff, and community leaders visit students’ homes to explain the 
program and encourage parents and students to participate. 

• Social services and academic enrichment through one of two programs — 
Communities In Schools (CIS) or the Campus Family Support (CFS) Plan 
(developed by Project GRAD) — which bring additional social services, 
academic activities, and volunteers into Project GRAD schools to address is- 
sues that students and their families face and to build commitment to aca- 
demic success. 

• Classroom management programs developed by Consistency Manage- 
ment & Cooperative Discipline (CMCD) SM that are designed to produce or- 
derly classrooms focused on learning, by promoting student responsibility 
and self-discipline and positive relationships among students, teachers, and 
other adults in the school. 
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Components at Project GRAD Elementary Schools 

During the time covered by this study, Project GRAD influenced instruction at elemen- 
tary schools directly by putting in place reading and mathematics curricula. Most Project 
GRAD sites used Success for All (SFA), a nationally recognized reading program that focuses 
on the key elements of reading instruction during concentrated instructional time (90 minutes 
each day), with the goal of bringing students to grade-level reading by third grade. Math Oppor- 
tunities, Valuable Experiences, Innovative Teaching (MOVE IT™ Math) was Project GRAD’s 
recommended math curriculum. It offers elementary school teachers professional development 
and instructional materials organized around the National Council of Teachers of Mathematics 
(NCTM) Standards program, involves heavy use of manipulatives to address a wide variety of 
learning styles, emphasizes daily problem solving, and introduces algebra in the early grades. 
(Currently, Project GRAD supports whatever reading and math curricula that participating dis- 
tricts adopt.) In addition, Project GRAD elementary schools also offer the three core compo- 
nents focused on parent and community involvement, social services and academic enrichment, 
and classroom management. 

In 1998, Newark, New Jersey, became the first site outside Houston to implement Pro- 
ject GRAD; Columbus, Ohio, and Atlanta, Georgia, followed soon thereafter. Currently, Project 
GRAD operates in five feeder patterns in Houston and in 12 school districts and 211 schools in 
eight states across the country, serving more than 131,000 students. To manage and support 
each Project GRAD initiative, local not-for-profit organizations were established in Houston 
and the expansion sites. 

Five important points should be noted about the implementation of Project GRAD in 
the three study sites — Houston, Atlanta, and Columbus: 

• The schools in this study did implement the core Project GRAD compo- 
nents and followed the general approach set forth in the model. How- 
ever, the Project GRAD strategy takes time to unfold; in each site, the 
various components at all schools were rolled out gradually. 

• Expansion within the Houston schools and to other school districts 
stretched the capacity of some program developers to support the 
model’s components and prompted the development of a national or- 
ganization in 2000 — Project GRAD USA — to sustain implementation 
efforts and to address implementation issues across sites. 

• The model’s theory presumes that feeder patterns are sufficiently “self- 
contained” so that better-prepared students would come to Project 
GRAD high schools from Project GRAD feeder schools. However, the 
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feeder patterns in the districts studied were more “leaky” than expected, 
in part as a result of high rates of student mobility and of policies that al- 
lowed students to choose the high school they attended. 

• This evaluation covers a period in which many efforts were being 
launched in the districts under study to improve student performance, 
including reforms specifically focused on high schools. Thus, “business 
as usual” — as represented by the experiences of the comparison schools 
— was neither static nor absent reforms. 

• The Houston experience provides the clearest opportunity to examine 
Project GRAD at the high school level, for several reasons: There was 
longer and stronger implementation; a greater percentage of students 
entered the high schools with Project GRAD exposure in earlier grades; 
and — because of Houston’s earlier start — there were fewer competing 
high school reform efforts. 

How Was the Evaluation Conducted? 

The goal of this evaluation is to understand whether Project GRAD changed the aca- 
demic outcomes of high school students from what they would have been absent Project GRAD 
— and, if so, how. To estimate the program’s effect on achievement, MDRC used an approach 
called “comparative interrupted time series analysis,” which compares the performance of Pro- 
ject GRAD schools with similar schools that have not implemented the reform. The first step in 
estimating program impacts with this design is to compare the change at Project GRAD schools 
in a given student outcome after the schools began implementing Project GRAD with the aver- 
age outcome during a baseline period, before implementation. This estimate represents how 
student performance changed in the presence of Project GRAD but does not, by itself, provide a 
measure of the effect of Project GRAD. The next step is to measure the corresponding change 
during the same period for similar schools in the same districts that were not implementing Pro- 
ject GRAD. This measurement provides an estimate of how student performance would most 
likely have changed at the Project GRAD schools had Project GRAD not been implemented. 
The difference between these two changes is an estimate of the impact of the Project GRAD 
reform — the effects that can be attributed to Project GRAD. 

This evaluation examines a number of student outcomes over varying numbers of years, 
depending on the site. For Houston, the study tracked attendance rates, test scores, promotion 
rates, credits earned, graduation rates, and the proportion of students completing a core aca- 
demic curriculum — for up to ten years at Jefferson Davis High School and its comparison 
schools and for seven or eight years at the two other Project GRAD high schools and their 
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matched comparisons. For the expansion sites in Atlanta and Columbus, data limitations al- 
lowed for only a maximum of three years of follow-up on two outcome measures: attendance 
rates and promotion rates. 

The very nature and complexity of the Project GRAD feeder system intervention, 
which posits that students would need to be exposed to the program over many years, combined 
with the limited amount of follow-up in the expansion sites, created a challenging set of condi- 
tions for a meaningful evaluation. While the Houston feeder patterns provide a reasonable test 
of the intervention, the results for Atlanta and Columbus should be treated as more provisional. 
In addition, because the high school evaluation began concurrently with the implementation of 
Project GRAD in the expansion site feeder elementary and middle schools, the study was not in 
a position to capture cumulative effects of students’ exposure to the intervention in the earlier 
grades in those districts. Only in the latter years of follow-up in Houston did this become possi- 
ble. Finally, findings from the single high schools in Atlanta and Columbus — as well as from 
Davis High School in Houston — should be interpreted with some caution. 



How Did Project GRAD Affect High School Student Outcomes? 

The findings from this study — combined with results from the companion report on 
elementary schools — provide insights into the strengths and limitations that Project GRAD 
brings to school improvement efforts as well as into the challenges that it still faces. MDRC’s 
elementary school evaluation found that students at Project GRAD schools generally showed as 
much improvement on high-stakes state achievement tests as students at similar local schools 
— but without suffering the decline seen in comparison schools on national tests, which meas- 
ure achievement more broadly. This suggests that Project GRAD can modestly improve the 
academic achievement of students leaving elementary school — an essential element of its 
strategy to develop better-prepared students entering high school. 

The findings from the high school study tell a more complicated story: 

• Focusing on early indicators of high school success (like credits earned 
and test pass rates in ninth and tenth grades), trends were generally 
similar and improving at the Project GRAD and comparison high 
schools in Houston, and any differences in the extent of improvement 
are not statistically significant. 

For example, average credits earned during ninth-grade and the percentage of students 
earning an algebra credit in ninth grade generally increased at both Project GRAD and com- 
parison schools. In addition, the percentage of ninth-graders at the Project GRAD and compari- 
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son schools who took and passed the tenth-grade reading and math portions of the state 
achievement test increased. 

• At Davis High School in Houston, increases in the percentage of ninth- 
graders who completed a core academic curriculum on time outpaced in- 
creases at the comparison schools. However, this effect did not hold up 
across the three original Project GRAD Houston high schools as a whole. 

Completing a core academic curriculum on time is defined in this study as graduating 
from high school within four years; earning four credits in English, three in math, two in sci- 
ence, and two in social studies; and maintaining an average grade of 75 out of 100 in core 
classes. During the baseline period at Davis High School, 9 percent of first-time ninth-graders 
met these requirements. In the years following Project GRAD’s implementation, the percentage 
of ninth-graders meeting these criteria at Davis increased by 12 percentage points, to 21 percent. 
This exceeds the progress at the comparison schools, which improved from 1 1 percent during 
the baseline period to 17 percent by the end of the follow-up period. The gain at Davis repre- 
sents a statistically significant positive effect of nearly 7 percentage points. When the results 
from Davis are pooled with the results from the other two high schools in Houston, however, 
improvements at Project GRAD and comparison schools are very similar. 

• Project GRAD does not appear to have had an independent effect on the 
percentage of ninth-graders who later graduated from a Houston high 
school. At both the Project GRAD and the comparison schools, the 
graduation rate slowly improved over the period of the study, but any 
differences between schools in the extent of improvement are not statis- 
tically significant. 

In this study, the graduation rate was defined as the proportion of first-time ninth- 
graders who graduated four years later. Although graduation rates improved at both Project 
GRAD and comparison high schools in Houston, the majority of ninth-grade students in any 
particular follow-up year did not graduate within the next four years. 

• Neither Atlanta nor Columbus has operated the program enough years 
to assess its effects on graduation rates or completion of a core academic 
curriculum, but early indicators suggest that Project GRAD may have 
had a positive influence on two precursors to graduation. 

The initial Project GRAD high schools in Columbus and Atlanta showed improvements 
in attendance and in promotion from ninth to tenth grade that appear to have outpaced improve- 
ments at comparison schools, though the differences are only sometimes statistically significant. 
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What Are the Implications of These Findings? 

What accounts for this pattern of findings? The evidence points to at least two hypothe- 
ses. First, it may be more difficult than expected to quickly improve the academic performance of 
incoming high school students by intervening in feeder schools — a core tenet of Project GRAD’s 
strategy. The implementation research highlights the fact that the Project GRAD strategy takes 
considerable time to unfold. In addition, it turns out that the feeder patterns for the high schools in 
this study were “leaky” — as a result of high rates of mobility and school-choice options, many 
students in the Project GRAD high schools had not benefited from exposure to the model in ele- 
mentary or middle school. For example, after ten years of program implementation in the Davis 
High School feeder pattern, the average pre-high school exposure to Project GRAD for incoming 
ninth-graders was approximately 4.5 years. In addition, by Year 6 of implementation across the 
three Houston high school feeder patterns, when ninth-grade students could have had up to five 
years of Project GRAD exposure in earlier grades, they had had only three years on average. 
Therefore, a considerable proportion of ninth-grade students in the Project GRAD high schools 
had not been exposed to Project GRAD in earlier grades, and among those students who had pre- 
high school Project GRAD exposure, many did not receive the full treatment. 

Second, other research has zeroed in on the crucial transitions that students must make 
as they enter high school, particularly on the central role that completing ninth grade on time 
plays in a student’s eventual completion of high school. 2 Being “on track” at the end of the first 
year of high school is a stronger predictor of eventual on-time graduation than a student’ s enter- 
ing achievement level is. 3 One explanation for the lack of impacts by Project GRAD on most 
ninth- and tenth-grade measures (and perhaps on graduation rates) could be the initiative’s lack 
of a direct intervention in ninth-grade instruction during the time period covered by the study. 
On the other hand, while Project GRAD did not produce improvements in graduation rates, its 
services were able to affect the course-taking of students at Davis High School who were al- 
ready headed toward graduation. The scholarship requirements, counseling on the best aca- 
demic preparation for college, and social service supports are likely reasons for the impacts on 
the academic preparation of graduates at the flagship high school in Houston. 

What do these findings mean for the future of Project GRAD? Given Project GRAD’s 
long-view strategy of developing better-prepared students in feeder schools, this evaluation of 
its high school intervention has come relatively early in its development, particularly in the ex- 
pansion sites outside its home district of Houston, where high school students would not have 
had the opportunity to participate in Project GRAD in earlier grades. At the same time, the mar- 

"Janet Quint, Meeting Five Critical Challenges of High School Reform: Lessons from Research on Three 
Reform Models (New York: MDRC, 2006); Melissa Roderick, Closing the Aspirations-Attainment Gap: Im- 
plications for High School Reform. A Commentary from Chicago (New York: MDRC, 2006). 

Roderick (2006). 
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ketplace in which Project GRAD operates is highly competitive. Local decision-makers, who 
face intense pressure to improve high schools quickly, are weighing the long-term Project 
GRAD strategy against other reforms that intervene more directly in the daily educational ex- 
perience of high schools — many of which have not taken on a comparative study like this to 
understand their own added value. 

One of the key implementation findings of MDRC’s study is that Project GRAD is a 
dynamic organization that has responded to operational lessons and research evidence to modify 
its strategy over time. In fact, it has already begun to refine its high school approach to address 
some of the challenges suggested by this evaluation, including the effects of “leaky” feeder pat- 
terns, of the relatively slow rollout of the components of the Project GRAD intervention, and of 
the difficulties of transforming high schools without directly intervening in the classroom. 

A recent study identifies “twin pillars” of successful high school reform efforts: person- 
alizing the learning environment and improving instruction . 4 Project GRAD’s high school inter- 
vention appears to focus more on the first pillar. To accomplish its goals of improving academic 
achievement, high school graduation rates, and rates of college attendance for low-income stu- 
dents, Project GRAD may need to make a strategic choice: to intervene directly in classrooms 
(on its own or in partnership with curricular reformers) or to target high schools where curricu- 
lar reform is already under way and where Project GRAD’s services and scholarship offer 
would provide added value. 

In many ways, Project GRAD's multifaceted strategy was ahead of its time, prefiguring 
a variety of current reform approaches. The focus on the full span of grades, the connection to 
postsecondary education, and the need to work above the level of individual schools are now 
appreciated as important aspects of many district-level reforms. Project GRAD now has the op- 
portunity to build on its strengths, incorporate additional components into its strategy to address 
its weaknesses, and develop strategic partnerships with other complementary school improve- 
ment efforts to create the next generation of its reform model. 



4 Quint (2006). 
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Chapter 1 

What Is Project GRAD? 



Introduction 

Project Graduation Really Achieves Dreams (GRAD) is an education reform initiative 
that targets a high school and the middle and elementary schools that feed into that high school. 
It combines a number of reforms with the objectives of increasing reading and math achieve- 
ment test scores, improving classroom behavior, providing a safety net for students to help re- 
duce dropout rates, and increasing rates of college enrollment and graduation for students at 
risk. The initiative originated at Jefferson Davis High School in Houston, Texas, and subse- 
quently expanded to other high school feeder patterns in Houston and in other school districts 
across the country, including the two expansion sites included in this evaluation: Atlanta, Geor- 
gia; and Columbus, Ohio. 1 This report provides an overview of Project GRAD’ s approach at the 
high school level and describes the experiences and findings at the first high schools to imple- 
ment Project GRAD in Houston and at the high schools in the early expansion sites. 

Project GRAD is intended to improve student outcomes and close the academic 
achievement gap between low-income and minority students and their more advantaged coun- 
terparts. The mission of Project GRAD is to ensure a quality public school education for all 
children in economically disadvantaged communities, so that high school graduation rates in- 
crease and students are prepared to enter and graduate from college. Specifically, Project 
GRAD has two goals: (1) to ensure that 80 percent of all entering ninth-graders in its high 
schools graduate and (2) to ensure that 50 percent of Project GRAD high school graduates go 
on to college. 

Originally built around a college scholarship offered to high school students who quali- 
fied, Project GRAD has evolved into a “feeder pattern” approach to improving education: Spe- 
cial attention is given to the elementary and middle schools that feed into an individual high 
school. In a clear break from attempts at systemic reform that assume that school-by-school 
transformations will eventually reach all children, Project GRAD targets an entire feeder pat- 
tern, seeks to be involved academically with students from preschool through high school (and 
for the four years of college), and aims to influence the school staff members whom students 

'Project GRAD USA (a national nonprofit organization that coordinates the initiative) now works with 12 
districts, including rural sites in Alaska, and anticipates that several additional districts will join the initiative 
soon. Currently, 211 schools and over 131,000 students are part of a network that includes 21 high schools, 34 
middle schools, and 149 elementary schools. (Alaska’s Kenai Peninsula has 7 schools that contain both ele- 
mentary and secondary grades.) Project GRAD Ohio is a special statewide effort in which lour of the state’s 
largest districts are implementing the initiative. 
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encounter from school to school and from grade to grade. The initiative has adopted this ap- 
proach for several reasons: 

• Project GRAD seeks to affect the prospects of large numbers of students and 
thus must move beyond small-scale reforms in which results might be seen 
as the product of unusual circumstances in a school or of the individual ef- 
forts of exceptionally talented principals and teachers. 

• High student mobility among schools in the districts where Project GRAD 
operates puts a premium on consistency in the educational approach across 
those schools. 

• Project GRAD is based on the ideas that early-grade curricular components 
will produce a core of solid basic skills that are needed for high school success 
and that successive cohorts of students who have stronger skills will both allow 
and push high schools to build challenging curricula. Over time, the students 
entering high school will be at grade level in reading and math and will be- 
come accustomed to high expectations; schools thus can expand their curricula 
to include more college-preparatory and advanced-placement offerings. 



The Project GRAD Theory of Change 

Although Project GRAD is ultimately a high school reform effort, many of its strategies 
focus on the lower grades, to prepare students to be more successful once they reach high 
school. At the elementary level, Project GRAD focuses on mathematics and literacy as two core 
components. MOVE IT ™Math (a K-8 program) is the preferred math program, and Success for 
All (SFA) is the reading program in most Project GRAD schools (Pre-K-5). 2 Combined with a 
classroom management component — Consistency Management & Cooperative Discipline SM 
(CMCD SM ) — and a social service/parental involvement component (either Communities In 
Schools [CIS] or the Campus Family Support [CFS] Plan developed by Project GRAD) that 
operate in all grades, the curricular components lay the foundation for anticipated longer-term 
results. (Box 1.1 explains Project GRAD’s operational strategies, and Box 1.2 describes its core 
components across all grades.) 

In its theory of change, Project GRAD hypothesizes that the model' s components at the 
elementary and middle school levels will create an improved educational climate and better 



2 SFA has also developed a middle school component that is used in some Project GRAD sites. 
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Box 1.1 



Project GRAD’s Operational Strategies 

• Creating a school and classroom environment that is conducive to learning. 

Project GRAD is typically introduced in schools that have discipline problems, weak 
student engagement, and a low sense of efficacy and pessimism about the possibility 
of academic success. Project GRAD feeder patterns are frequently described as 
those serving the lowest-performing high schools and/or many of the most troubled 
elementary schools. In response, Project GRAD seeks to address students’ social 
service needs and strengthen school-community linkages (through Communities In 
Schools [CIS] or Campus Family Support [CFS]). promote student self-discipline 
and engagement (through Consistency Management & Cooperative Discipline SM 
I CM CD SM I), and create excitement about the possibility of going to college 
(through a scholarship offer and summer institutes on college campuses and special 
Project GRAD activities such as the Walk for Success). The annual Walk for Suc- 
cess — which includes visits to the homes of students in the feeder pattern by school 
staff and community representatives — is sponsored each year - by the local Project 
GRAD office to build awareness of the scholarship offer and the goals of the initia- 
tive and to encourage parental involvement. Both CIS (or CFS) and CMCD SM are 
offered at all grade levels. Any high school student who meets Project GRAD’s re- 
quirements can receive a scholarship, which is publicized in elementary and middle 
schools as well, to help improve students’ (and teachers’) motivation and to foster a 
sense of the possibility of real change. 

• Strengthening opportunities to learn. Low student achievement is often one rea- 
son that a district or group of schools adopts Project GRAD. Typically, an early goal 
of the initiative is to strengthen instruction in the lower grades, to help improve the 
basic academic skills of students and to prepare them for the secondary grades. Key 
Project GRAD components focus on reading (typically, Success for All [SFA]) and 
mathematics (preferably, MOVE IT ™ Math) in grades 1 through 6, to provide stu- 
dents with the skills they need to succeed in other subjects and in later grades. SFA 
has pilot-tested a middle school program in some Project GRAD sites, and MOVE 
IT™ Math has expanded to the middle school grades. The use of these middle 
school curricular components has varied from site to site. The summer institutes for 
high school students (some of which are residential) also provide an opportunity to 
address specific skills gaps and to enrich the educational offerings in the Project 
GRAD schools. 

(continued) 
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Box 1 .1 (continued) 



• Enhancing school-level capacity for management and implementation. In each 
Project GRAD site, a local nonprofit organization is formed to support the imple- 
mentation of the program through the feeder pattem(s) and to coordinate with the lo- 
cal district. Project GRAD works with the existing students, school staff, and organ- 
izational structure; it does not require that the district reassign, alter, or adjust basic 
assets to implement the initiative. The local organization works with staff at the dis- 
trict and school-building level to implement the various components effectively, and 
it plays a key role in building local support for the initiative, by developing strategic 
partnerships with other institutions and by fundraising. Project GRAD’s outreach ac- 
tivities help foster higher community expectations for reform and school improve- 
ment. Although the degree to which the local organizations are connected to the 
school district varies, a partnership with the district is an important element in the 
Project GRAD implementation strategy, even though the local organizations are 
viewed largely as independent catalysts supporting improvement in the Project 
GRAD schools. The national Project GRAD organization — Project GRAD USA 
— provides technical assistance, coordinates component service delivery, develops 
new components and approaches, and provides funding support to the local organi- 
zations. Project GRAD USA also regularly convenes the Project GRAD network of 
districts and administrators that are implementing the initiative. 



When districts do not implement SFA or MOVE IT™ Math in Project GRAD feeder patterns. 
Project GRAD endeavors to support or supplement the district’s curriculum choice. Typical read- 
ing support, for example, might include reading consultants or a reading manager for each school, 
along with special training and materials. 



classroom instruction . 3 Indicators of success in inducing the intended changes in students’ edu- 
cational experiences include reduced discipline referrals and suspensions, improved attitudes 
toward school, higher expectations, improved attendance, greater time on task, and increases in 
the school’ s offering and the students’ completing more demanding courses. These changes, in 
turn, are expected to lead to improved student achievement in elementary and middle grades, as 
indicated by higher achievement test scores and greater numbers of students performing at 
grade level. Such improvements in achievement are also expected to further improve the 
schools’ climate and instruction. The feeder-based activities of Project GRAD — its academic 



3 See Project GRAD USA (2004b, 2004c). 
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Box 1.2 



The Core Components of Project GRAD 

• The Project GRAD college scholarship. The program’s offer of a college scholar- 
ship is often referred to as the cornerstone of Project GRAD because it most directly 
represents the ultimate goal of the initiative: increasing student enrollment and suc- 
cess in college. Through this component, Project GRAD seeks to raise the academic 
expectations of students in grades K- 1 2 by providing a financial incentive and col- 
lege awareness, preparation, and recruitment activities. Scholarship amounts and cri- 
teria vary slightly by site but usually average $1,000 to $1,500 each year during the 
four years of college. Project GRAD scholars are high school youth who sign a con- 
tract to document their intent to meet the program’s scholarship criteria. 1 The Project 
GRAD scholarship coordinator is a critical part of the high school features of the ini- 
tiative. The coordinator serves as a college access counselor for potential Project 
GRAD scholars and provides counseling, tutoring, college awareness, and college 
admission preparation. 

• Summer institutes. This component provides an opportunity for Project GRAD 
scholars to experience a college-campus-based program taught by college faculty 
and to enhance their academic skills. The institutes vary by site but typically include 
reading, writing, math, science, and enrichment and — as needed — remedial activi- 
ties. The summer institutes typically include four to six hours of instruction and re- 
lated activities per day for four to six weeks. 

• Classroom management. Consistency Management & Cooperative Discipline^ 
(CMCD SM ) is a classroom management and discipline program that seeks to build 
student support for classroom management, responsibility, and self-discipline by 
promoting cooperative learning and positive working relationships among students, 
teachers, and other adults in the school. It moves beyond the traditional concept of 
student discipline as a set of adult responses to a student’s negative behavior to offer 
a comprehensive classroom management program that is intended to build a shared 
sense of responsibility among students and adults in the building. CMCD SM is put in 
place in all grades at the Project GRAD schools.* 

• Social services and parental involvement. Communities In Schools (CIS) is a na- 
tional program that brings additional support (that is, volunteers, social services, and 
academic emichment and support activities) directly into schools. It is implemented 
in all grades at Project GRAD schools. In some sites, each school has a full-time CIS 

(continued) 
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Box 1.2 (continued) 



staff person. This component seeks to enhance social and academic support services 
that are available through the school and to provide targeted assistance to students 
who have problems outside school that may affect their classroom performance. A 
key objective of Project GRAD’s focus on community involvement is to build fam- 
ily aspirations for academic success and college -going. The range of activities may 
include guidance counseling, community outreach, and family case management. 
For school districts that do not have an existing CIS program. Project GRAD USA, 
the national coordinator for the initiative, has developed the Campus Family Support 
(CFS) Plan, which contains similar elements. Coordinated by staff from either CIS 
or CFS, Project GRAD conducts an annual Walk for Success, during which stu- 
dents’ homes are visited by the program’s staff, district staff (including principals 
and teachers), and community leaders, who explain Project GRAD and the scholar- 
ship offer and encourage parents to connect with their child’s school. 

• A research-based literacy program. A reading program is put in place in the Project 
GRAD elementary schools. Most but not all sites use Success for All (SFA), a nation- 
ally recognized reading program that promotes comprehensive restructuring of most 
school resources to provide concentrated instructional time for reading (90 minutes per 
day), to bring students to grade level in this subject area by the third grade. 

• A research-based math curriculum. This component is also put in place in the 
elementary schools. Project GRAD’s preferred math curriculum is Math Opportuni- 
ties, Valuable Experiences, Innovative Teaching (MOVE IT™ Math), which is a K- 
8 professional development program that advocates math instruction based on the 
use of manipulatives to address a wide variety of learning styles. It emphasizes daily 
problem solving and introduces algebra in the early grades. 

In some sites. Project GRAD also operates in pre-K classes. 

'Project GRAD’s college scholarships are for students who have a cumulative 2.5 grade point av- 
erage, graduate within a four-year time period, complete a recommended college preparatory cur- 
riculum, and participate in two summer institutes that are designed to expose students to an en- 
hanced curriculum while introducing them to college. 

“Project GRAD USA now has an agreement with CMCD SM that allows it to manage the compo- 
nent at each site after two and a half years of implementation. 
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support, social services, parental involvement, and community outreach — are geared to moti- 
vate students and to build the community’s expectations for school reform. 4 

The momentum created by improved student achievement at the lower grades is hy- 
pothesized to drive up achievement at the high school level as well, particularly as better- 
prepared students’ performance in existing courses improves and creates demand for higher- 
level courses. Better-prepared students also result from the improved motivation caused by Pro- 
ject GRAD’s scholarship offer, which is emphasized to students and their parents beginning in 
the earliest grades. When students reach the high school level, the offer becomes a more imme- 
diate reward: the opportunity to earn a scholarship to go to college. Improved student achieve- 
ment then produces more students who are able to complete a college preparatory curriculum 
and more who complete high school, go on to college, and earn a postsecondary degree. Figure 
1 . 1 presents the sequence of high school effects that Project GRAD intends to produce. 



Description of the Project GRAD High School Program 

At the high school level, Project GRAD consists of the following components: 5 

• The college scholarship offer. To make college enrollment and completion 
more realistic for low-income students and their families — who may have had 
little exposure to college — the Project GRAD college scholarship program of- 
fers high school graduates $1,000 to $1,500 (depending on the Project GRAD 
site) for each of the four years of college. The scholarship offer is often re- 
ferred to as the cornerstone of Project GRAD, because it most directly repre- 
sents the ultimate goal of the initiative: increasing students' enrollment and 
success in college. Students and their families in the elementary and middle 
schools are made aware of the scholarship offer, and it is frequently used as a 
motivational tool and combined with other activities to foster college aware- 
ness. At the high school level, efforts to emphasize the scholarship are intensi- 
fied. Ninth-graders at Project GRAD high schools can choose to sign a “con- 
tract” expressing their commitment to strive for the scholarship, and they are 



4 For a description of the development of Project GRAD, see Holland (2005). 

5 For additional information about the program’s components at the elementary school level, see Snipes, 
Holton, and Doolittle (2006). In addition. New Site Development Handbook (Project GRAD USA, 2004a) de- 
tails the program’s components and outlines the process and requirements that potential sites must follow in 
order to become a Project GRAD site. The handbook includes information about the high school program and 
which elements need to be put in place to support it. 
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then designated as “Project GRAD scholars.” To be eligible for the scholar- 
ship, these students must graduate on time from a Project GRAD high school, 
have at least a 2.5 grade point average, attend at least two summer institutes, 
and complete the district’ s college preparatory courses. 

• The scholarship coordinator. The coordinator works with the Project 
GRAD scholars to ensure that they are on track for graduation and that they 
meet the scholarship criteria. The coordinator also helps encourage students 
to sign up for the scholarship and identifies additional scholarships and fi- 
nancial aid that might be available. The social services and parental involve- 
ment component of Project GRAD (described below) also provides a range 
of additional supports for the scholars, such as tutoring, career counseling, 
and college awareness activities. Unless a scholar enrolls in the military, he 
or she has only one year after graduation to enroll in a college, a university, 
or an accredited degree-conferring institution. 

• The summer institutes. Targeted to Project GRAD scholars in the tenth and 
eleventh grades, the summer institutes are held on college campuses, and 
classes are usually taught by college faculty. Some sites have offered residen- 
tial programs. For four weeks, Project GRAD scholars are exposed to a range 
of college experiences, and — depending on their individual needs — they 
receive remediation and/or academic enrichment. The institutes’ curricula 
vary by site but typically include reading, writing, math, science, and en- 
richment activities. Some summer institutes provide stipends, and some have 
been coordinated with summer work programs. In some sites, participating in 
other established summer programs that are also focused on college prepara- 
tion may be substituted for the Project GRAD summer institutes. 

• The social services and parental involvement component. Initially, Pro- 
ject GRAD's service component was Communities In Schools (CIS), a na- 
tional program that brings additional support (including volunteers, social 
services, and academic enrichment activities) directly into schools. Typically, 
CIS is implemented in all grades at Project GRAD schools and seeks to en- 
hance their social and academic support services and to target assistance to 
students who have problems outside school that may affect their classroom 
performance. Project GRAD USA — the national coordinator of the initia- 
tive — has developed the Campus Family Support (CFS) Plan, which in- 
cludes similar elements for school districts that do not have a CIS program. 
CIS or CFS plays a critical role at the high school level, where such support 
services as mentoring, tutoring, and special activities and positive connec- 
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tions with a caring adult can all serve as elements of an important safety net. 
Typically, each Project GRAD high school is assigned a full-time CIS or 
CFS staff person, and some schools have additional staff. 

• The classroom management component. Consistency Management & Co- 
operative Discipline SM (CMCD SM ) seeks to build student support for class- 
room management, responsibility, and self-discipline by promoting coopera- 
tive learning and positive working relationships among students, teachers, and 
other adults in the schools. CMCD SM is put in place in all grades at the Project 
GRAD schools. At the high school level, discipline and safety issues are fre- 
quently serious concerns, and — by setting a code of conduct and establishing 
uniform procedures for how classrooms operate — CMCD SM seeks to ensure 
that disruptive behavior does not take time away from instruction. 

• The Walk for Success. Project GRAD views parental involvement as a 
critical ingredient in the success of students and schools. Recognizing that an 
integral part of making college “real” to young people is working with par- 
ents to ensure that they are engaged in their child’s schooling and that they 
fully understand the scholarship opportunity, each Project GRAD program 
holds an annual Walk for Success, which is usually spearheaded by CIS or 
CFS staff. On a designated day, volunteers visit the homes of families in the 
school’s neighborhood to explain the Project GRAD initiative and its schol- 
arship offer and to encourage students in the appropriate grades to sign up. 
Typically, the Walk for Success becomes a galvanizing force for the school 
community, and it is not uncommon for elected officials, major funders, 
members of parent-teacher organizations, community leaders, and the dis- 
trict’s leadership and school’s staff to participate. In addition to securing a 
signed commitment from parents and students, volunteers conduct a survey 
of parents to learn which information, workshops, or continuing education 
classes they would like to have to help them better educate their children. 

Against this backdrop of program components, the “scholars” in a typical Project 
GRAD high school are periodically convened for special activities and for motivational support, 
and, in many regards, they become an identified group within the school. Their grade point av- 
erages are routinely checked; the forward progress of individual students is celebrated; and stu- 
dents and their families are kept informed of their standing with regard to meeting graduation 
requirements and the scholarship criteria. 

At times, Project GRAD has coexisted or worked collaboratively in the same high 
school building as other reform models or approaches, including block scheduling, grade-level 
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academies, the Talent Development High School model, the Gates small schools initiative, and 
High Schools That Work. However, the high schools in the Project GRAD network have no 
required curricular components that are elements of the initiative . 6 Over time, individual sites 
have implemented a range of enhancements at the high school level, including technical assis- 
tance to the schools, professional development for teachers, and new academic offerings. 

High schools in the Project GRAD network also benefit from the structural components 
that are integral to the initiative’ s approach: 

• As the initiative is phased in and students move from grade to grade, the high 
school in the feeder pattern will receive students who, over time, have bene- 
fited from Project GRAD’s curricular components in the early grades . 7 

• Project GRAD high schools receive technical support from both the local and 
the national Project GRAD organizations, benefiting from their efforts to fos- 
ter community involvement and constituency support for school reform and 
to collaborate and build relationships with business, funding, and higher edu- 
cation partners. 

• The initiative’ s commitment to work with the students, teachers, and admin- 
istrators who are already in the school building minimizes disruption and al- 
lows for a faster transition to Project GRAD, since neither students nor 
teachers need to be reassigned. 



Key Questions Addressed in This Report 

This report grows out of MDRC’ s independent third-party evaluation of Project GRAD. 
The evaluation initially focused on the early expansion sites but eventually came to include a 
retrospective analysis of the early feeder patterns in Houston, the flagship district. The compan- 
ion report from this evaluation analyzes the effects of Project GRAD on elementary students’ 
outcomes . 8 The present report focuses on Project GRAD’s high school components and what 



6 Project GRAD has recently initiated an enriched academic high school model (currently in the pilot- 
testing phase) that includes (1) a summer program for students who are about to enter the ninth grade, to 
strengthen their math and literacy skills and to introduce them to the requirements of high school; (2) remedial 
and/or advanced work in literacy or math during the school year; and (3) pre-advanced placement courses, 
advanced placement courses, and support activities for such examinations as the American College Test 
(ACT), the Preliminary Scholastic Assessment Test (PSAT), and the Scholastic Achievement Test (SAD- 

7 Typically, when Project GRAD begins implementation, the cohort of entering ninth-graders is eligible for 
the scholarship, even though these students were not exposed to the program’s components in the elementary 
and middle grades. 

8 See Snipes, Holton, and Doolittle (2006). 
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has been learned so far about the program’ s effects on student outcomes at three high schools in 
Houston and at one high school each in Columbus and Atlanta, two of the expansion sites. 9 

The evaluation focuses on the following questions: 

• How do key high school student outcomes change at Project GRAD high 
schools in the years following the program's implementation? 

• How do the changes in student outcomes at Project GRAD high schools 
compare with the changes at similar high schools from the same districts 
over the same time period? 

Inasmuch as the period of Project GRAD’s implementation in Houston permits a 
longer-term analysis than is possible for the expansion sites, the evaluation focuses on a variety 
of outcomes, from early indicators of high school success (such as ninth-grade attendance and 
ninth- to tenth-grade promotion and the earning of algebra credits) to ultimate indicators of suc- 
cess (such as high school graduation rates and the completion of a core college preparatory cur- 
riculum). Because follow-up data in the expansion sites are more limited, the analysis focuses 
on early outcomes in high school as precursors to graduation. 



Topics Addressed in This Report 

This report builds on and extends to the high school level the descriptions of the Project 
GRAD model that are presented in MDRC’s companion report on the implementation of Pro- 
ject GRAD at the elementary school level in Houston and in the early expansion sites. Chapter 2 
presents Project GRAD’s evolution and the implementation experience at Jefferson Davis High 
School in Houston, to provide information on how and why the reform was developed and on 
changes in the first site, which led to the subsequent expansion efforts. Chapter 2 also describes 
the implementation efforts at high schools in the expansion sites, Columbus and Atlanta. Chap- 
ter 3 describes the analytic approach to the study and presents the estimated effects of the pro- 
gram on student outcomes, including graduation rates, at the Project GRAD high schools in 
Houston. Chapter 4 describes the estimated program effects on early indicators of high school 
success in Columbus and Atlanta. Chapter 5 concludes the report by summarizing the findings 
across sites and placing them in the broader context of education reform and efforts to improve 
outcomes for high school students. 



9 The initial expansion sites were Newark, NJ; Columbus, OH; Los Angeles, CA; Nashville, TN; and At- 
lanta, GA. Los Angeles is not included in this analysis because the district’ s data system does not provide the 
historical information required by the study’s methodology. Nashville is no longer a Project GRAD site. The 
lack of appropriate and reliable data prevents similar analyses for Newark, the first expansion site. 
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Chapter 2 



The Implementation of Project GRAD 
at the High School Level 

The Evolution of the Project GRAD Model 

Although the effort to reform what was to become the first Project GRAD high school 
in Houston dates back to a 1 982 partnership between the Houston Independent School District 
(HISD) and the business community, the comprehensive reform framework that became Project 
Graduation Really Achieves Dreams (GRAD) did not emerge until the 1993-1994 school year. 
The initiative was first implemented in the Jefferson Davis High School feeder pattern — that 
is, a cluster of elementary and middle schools that feed into the high school; these schools pri- 
marily serve Hispanic students. Based on encouraging results in this first feeder pattern, Project 
GRAD expanded to other feeder patterns within the HISD. Currently, five of the district’s 
feeder patterns use Project GRAD as their reform strategy. 

The educational challenges that confronted Jefferson Davis High School prior to Project 
GRAD were considerable. 1 An assessment using the Iowa Test of Basic Skills (ITBS) indicated 
that 50 percent of the 1982-1983 graduating class were performing at more than four years below 
grade level. 2 The Jefferson Davis Educational Collaborative — the precursor to Project GRAD — 
was spearheaded by the CEO of Tenneco, James Ketelsen, with the hope of improving students’ 
interest in school, their academic performance, and the proportion of graduates who attended col- 
lege. Initial elements of this early program included the provision of adult mentors and tutors for 
students and awards of $2,500 per year for two college-bound graduating seniors. 

In school year 1983-1984, Tenneco’ s commitment to the reform effort facilitated the 
addition of on-site social support services through Communities In Schools (CIS) and a summer 
jobs program for students who were at risk of dropping out of school. In 1985, an annual sum- 

'On the eve of Project GRAD’s initiation in the 1993-1994 school year, levels of academic performance 
and discipline in the Davis feeder pattern's middle schools and high school were generally very low. For in- 
stance, among sixth-graders at Marshall Middle School who took the Norm-Referenced Assessment Program 
for Texas (NAPT), only 11 percent scored at or above the 50th percentile of national scores in reading, and 
only 20 percent did so in mathematics. Among eleventh-graders at Davis High School, 10 percent scored at or 
above the 50th percentile on the NAPT in reading, and 36 percent did so in math. In addition, Marshall Middle 
School — with a total enrollment of 1,100 students — experienced 300 out-of-school suspensions, 13 expul- 
sions, and 280 grade-level retentions (24 percent) in 1993-1994. Furthermore, it was ranked lowest of the 
HISD's 34 middle schools on the state’s basic skills test, the Texas Assessment of Academic Skills (TAAS). 
Such low performance could not be blamed entirely on instruction at the secondary level but also reflected 
curriculum deficiencies in the feeder system’s elementary schools (Opuni, 1999). 

2 Opuni (1995). 
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mer program in leadership training and character development was initiated for selected Davis 
High School students. In 1988-1989, the scholarship component was expanded to provide an 
annual $1,000 college scholarship — sponsored by Tenneco — for four years for each college- 
bound graduating senior who met the program’s eligibility rules. 

When the college scholarship offer did not generate a substantial increase in high school 
graduation and college attendance, Ketelsen expanded the effort further, to include the associ- 
ated elementary and middle schools that fed into Davis High School and to combine additional 
reforms: a math and a literacy component in the lower grades and — in all grades — a class- 
room management program and the social services and community involvement component. 
This combination of reforms was intended to increase the chances that students could take ad- 
vantage of the opportunity provided by the college scholarship. 

Importantly, Project GRAD breaks away from the school-by-school approach used by 
many reform models and instead targets feeder patterns. In so doing, the initiative seeks to im- 
plement a set of reforms that follow students from the elementary grades through the end of 
high school. The Project GRAD model simultaneously offers support to students at all grade 
levels, even if they have not had the benefit of the initiative in their earlier years. In an effort to 
“work with the existing assets,” Project GRAD is implemented with the students, administra- 
tors, and teachers who are already in the schools. 3 

As with many reform strategies, Project GRAD' s principles were crafted in fairly broad 
terms at the outset and then were refined to reflect early operational experiences and lessons. 
The model is not based on one specific educational philosophy or pedagogical approach but, 
rather, combines existing program components and complementary operational strategies that 
are backed by research and/or experiential support. As such, the initial choice of components 
was more pragmatic than theory -driven, as compared with other reform approaches. As Project 
GRAD has matured, it has continued to develop and refine both its general approach and the 
individual components of the model. 



Milestones in Project GRAD’s Implementation 

Jefferson Davis High School officially became the flagship of the Project GRAD initia- 
tive in 1993-1994. 4 The Tenneco scholarship offer was in place as the core component, and CIS 
— the social service and family support component — was also in place. 5 Consistency Man- 

3 See Project GRAD USA (2004b, 2004c). 

4 To capture the point at which all the elementary schools were implementing a Project GRAD component, 
the analysis uses 1994-1995 as the first follow-up year. 

5 The scholarships were in place since 1988-1989. CIS began at Davis in 1983-1984, but funding from Pro- 
ject GRAD began in 1995-1996 (Opuni and Ochoa, 2004). 
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agement & Cooperative Discipline SM (CMCD SM ), a classroom management program developed 
by the University of Houston-Central Campus, was pilot-tested at the Project GRAD elemen- 
tary schools earlier, but it did not begin at Davis High School until 1996-1997. 

Project GRAD began in the Jack Yates High School feeder pattern, which primarily 
serves African-American students, with the announcement of the scholarship offer in the 1996- 
1997 school year. Initially, the Yates scholarships were called the Conoco and the James P. 
Grizzard Scholarships. Although CIS began in 1999-2000 6 for the entire Yates feeder pattern, 
the high school itself has not implemented CMCD SM . Phillis Wheatley High School was the 
third high school to implement the initiative; it announced the Project GRAD scholarships in the 
1997-1998 school year, and the other components of the model followed over the next several 
years. CIS began at Wheatley High School in 1999-2000, and CMCD SM began in 2002-2003. 

Project GRAD is coordinated by the Houston-based Project GRAD USA, a not-for- 
profit organization that came into existence as the initiative’s expansion effort began to gain 
momentum. This national organization is separate from Project GRAD Houston, the local not- 
for-profit organization that manages implementation in Houston. 

The implementation of Project GRAD in other school districts began in the 1998-1999 
academic year as the initiative expanded to Newark, New Jersey — and, in quick succession, to 
Los Angeles, California; Columbus, Ohio; Nashville, Tennessee; and finally Atlanta, Georgia. 
This report focuses on the expansion efforts in Columbus and Atlanta as a means of highlighting 
the opportunities and challenges of replicating the Project GRAD model in other school districts. 7 



Project GRAD Columbus 

Project GRAD Columbus began in the fall of 1999, a time frame in which two other ex- 
pansion sites, Los Angeles and Nashville (which are not part of this study), also began implement- 
ing the program. Coupled with the addition of new feeder patterns in Houston, the expansion sites 
placed added stress on the capacity of Project GRAD Houston and its component providers to 
support local implementation. Lacking today’s tools designed by Project GRAD USA to guide 
new sites in the implementation process, Columbus had to rely on the experiences of other sites 
and on visits to Houston to understand how to launch the initiative. 8 The feeder pattern in Colum- 
bus consists of 12 schools: a high school, 4 middle schools, and 7 elementary schools. 



6 The Project GRAD scholarship for the Yates feeder pattern was announced in the 1996-1997 school year. 
CIS received support from Project GRAD beginning in 1999-2000 (Opuni and Ochoa, 2004). 

7 The choice of expansion sites included in this report was driven by the availability of the historical data 
required for the analysis. 

8 See New Site Development Handbook (Project GRAD USA, 2004a). 
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Project GRAD’s emergence in Columbus coincided with intense community pressure 
to keep open the doors of what would become the city’s first Project GRAD high school. De- 
spite a rich history in the city, Linden-McKinley High School struggled with declining enroll- 
ments and many of the issues endemic to urban schools. A new principal and administrative 
team were brought in; staff members were reconstituted; the building was renovated; and the 
high school and its feeder schools were designated as Project GRAD schools in the fall of the 
1999-2000 school year. The maximum enrollment capacity of Linden-McKinley High School 
was about 1,500 students, but it enrolled only a little more than half. The student body, although 
largely African-American, includes many refugees from Africa — most notably, from Somalia. 

Based on students’ low scores, the school district in Columbus was identified by Ohio’s 
statewide performance accountability system as a district in “academic emergency.” This desig- 
nation requires a district to develop a continuous improvement plan, and the district is subject to 
monitoring and intervention by the state’s Department of Education and must adhere to annual 
progress guidelines. Columbus is also a district that for many years operated under a citywide 
school desegregation plan that resulted in considerable reassignment of students. 

Project GRAD was brought to the Columbus school superintendent by a local philan- 
thropist who became the chair of a small Project GRAD board and a key voice in developing 
the local not-for-profit organization. As it developed the relationship with the school district, 
Project GRAD Columbus benefited from the strength of its sister organization, I KNOW I 
CAN™ (IKIC), which was created in 1988 with the mission of encouraging students to stay in 
school and pursue a college education. 9 However, Project GRAD Columbus struggled to de- 
velop its own organizational framework. Staff were lent from the district or had dual roles in the 
district and, in later years, were shared with IKIC — all of which, at times, resulted in compet- 
ing priorities. Project GRAD scholars in Columbus have the opportunity to receive support both 
from the program and from IKIC, and the work of IKIC’s volunteers in the high school com- 
plements the mission of Project GRAD. 10 At the same time, Project GRAD has had to address 
the challenge of positioning the initiative as the effort of a distinct organization. 

“Student advocates” from the Communities In School (CIS) program were first placed 
in Linden-McKinley High School in the fall of 1999. Although CIS existed in Columbus prior 
to Project GRAD, it was a small organization that brokered services, rather than providing ser- 
vices directly to students in schools, as envisioned in the Project GRAD model and as has tradi- 
tionally occurred in Houston. Student advocates play many roles in their respective buildings, 

9 IKIC provides Last Dollar Grants (up to $1,200 annually) for college to every qualified graduate of the 
Columbus School District. 

10 IKIC volunteers are present at Linden-McKinley High School, where they make classroom visits to pro- 
mote postsecondary education options and counsel students individually in preparation for the college applica- 
tion process. Volunteers also recruit students for the summer institutes. 



16 




but their principal goals include reducing the rates of absenteeism and tardiness and increasing 
the level of connections between home and school. At times, the heavy emphasis on student 
attendance constrained the efforts of CIS in other areas. A strong theme in Project GRAD’s 
early implementation period related to concerns about the CIS organization’s ability to provide 
staff training and needed administrative support for the student advocates and for its ability to 
provide services within the context of Project GRAD. Because CIS Columbus had not hereto- 
fore managed a large staff nor deployed staff to schools, Project GRAD Columbus played a sig- 
nificant role in guiding the orientation, planning, training, and monitoring functions for CIS ad- 
vocates in Project GRAD schools. The Walk for Success (see Box 1.1 in Chapter 1) — with 
CIS staff playing a key role — has been held successfully three times and has received favor- 
able attention from the city, the district, and the local media. 

In January 2001, the implementation of Consistency Management & Cooperative Dis- 
cipline SM (CMCD SM ) began in three of four middle schools and in the high school. Support for 
CMCD SM was reasonably strong at the high school in the beginning stages of implementation, 
but enthusiasm waned somewhat due to what staff described as the prevalence of “elementary 
school” strategies in this component and in the programming for staff development. As a result, 
Project GRAD Columbus developed methods to address these issues and to strengthen imple- 
mentation, coupled with refinements by the developer. 11 

A summer institute that was pilot-tested for students in the summer of 2001 at Ohio 
State University was able to attract only a small number of students, followed by a larger pro- 
gram the next summer. The important position of scholarship coordinator was not filled until 
the fall of 2001. 

The student assignment plan in Columbus differs significantly from the feeder pattern, 
or “pipeline,” model by which Project GRAD is usually organized in Houston. The Columbus 
district has an “open enrollment” policy, which means that students from Project GRAD ele- 
mentary and middle schools do not necessarily feed into Linden-McKinley High School. 12 As a 
result, students who are enrolled at the four Project GRAD middle schools may apply to attend 
any high school in the district. Therefore, implementation of CMCD SM — a key intervention of 



u For a description of how the CMCD SM component was refined, see Holland (2005). 

12 Parents may elect to send their children to schools other than the assigned “home school” by participat- 
ing in a lottery — held each February for the following school year — for both conventional and alternative 
schools. There are six community centers serving geographical regions within the district. Parents may enter 
the lottery to send their children to schools in the region where they live (under the “Community Choice” pro- 
gram) or to schools outside the region (under the “intradistrict Open Enrollment” program). Students also have 
the option of attending one of the district’s four alternative high schools, four alternative middle schools, 
twenty alternative elementary schools, and two K-8 language-immersion schools. The district provides trans- 
portation for students who live more than 2 miles from their school. 
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the program at the high school — is affected both by the level of its implementation at the Pro- 
ject GRAD middle schools and by the inclusion of high school students who came from non- 
Project GRAD middle schools and thus were not exposed to CMCD SM earlier. This is particu- 
larly important, because CMCD SM is the anchor component of Project GRAD at the high 
school, which has implemented only CIS and the scholarship program. 

Although there is strong community support for Linden-McKinley High School — and 
support has grown since a new principal and staff were put in place — it is not uncommon for 
students who live nearby to attend other high schools. Most important, not all students who 
eventually attend Linden-McKinley will have had the benefit of the program’ s components in 
the lower grades and, consequently, will not have the full Project GRAD experience. Faced with 
the demand to rapidly improve the high school' s academic climate, the principal endorsed Pro- 
ject GRAD and its systemic theory of change (in which the most intensive efforts are at the 
elementary school level) but also recognized the need to broaden the academic improvement 
strategies at the high school. 

Finally, the experience of Project GRAD Columbus in attempting to expand to a second 
feeder pattern illustrates the difficulties confronting the site. In 2002, Project GRAD held an 
initial vote on whether to implement the program throughout the Mari on -Franklin High School 
feeder pattern. Though most of the schools did vote to adopt the program, enough of them did 
not, and the measure did not pass. Marion-Franklin rejected Project GRAD. However, in Janu- 
ary 2003, Columbus subsequently added Marion-Franklin High School and 10 elementary and 
middle schools as its second feeder pattern. 



Project GRAD Atlanta 

Project GRAD was introduced to Atlanta by the district’s new superintendent, who had 
supported the initiative’ s inception while serving as the state-appointed superintendent in New- 
ark, New Jersey. The superintendent’ s vision of reforming Atlanta Public Schools included the 
expansion of Project GRAD to additional feeder patterns and the district’s eventual takeover of 
program operations. This was underscored by including the executive director of Project GRAD 
Atlanta (the not-for-profit organization that manages the site’s implementation) in executive 
cabinet meetings. The first feeder pattern was selected based on a combination of two factors: 
choosing the most needy set of schools, as determined by low test scores, and choosing a set of 
schools that had strong leadership at the high school level. The Booker T. Washington High 
School feeder system was one of the lowest-performing academically but was perceived as hav- 
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ing strong leadership at the high school. The feeder pattern consists of a high school, nine ele- 
mentary schools, and two middle schools. 13 

Booker T. Washington High School predominantly serves African-American students 
and was the first all-black high school in the Atlanta Public Schools system. For many years, it 
was seen as the best African-American high school in the system, enjoying a strong reputation for 
student achievement and having principals who were revered. The high school has a rich history; 
the notables who either attended or graduated from it include Dr. Martin Luther King, Jr.; Dr. 
Louis Sullivan, former Secretary of Health and Human Services; and Robert Gibbs, the first Afri- 
can-American to enter and graduate from Harvard University. The high school is also near Atlanta 
University Center, where the following Historically Black Colleges and Universities are located: 
Clark Atlanta University, Morehouse College, Morris Brown College, and Spelman College. 

In a major push to combat students' poor academic performance, Booker T. Washington 
High School began implementing a number of special programs prior to Project GRAD, including 
Students for Teaching Careers and Academic Readiness (STAR), the Business Technology Cen- 
ter, and the Culinary Arts Academy (with a noted student-run restaurant). The high school also 
has a health clinic. Offerings in advanced placement courses were expanded to include Literature, 
History, Chemistry, Biology, Calculus, and Pace Setter courses (which feed into advanced place- 
ment courses) in math, English, and Spanish. Washington High School and the other comprehen- 
sive high schools in Atlanta were also part of the High Schools That Work initiative. 

Communities In Schools (CIS) officially began at the Project GRAD Atlanta schools in 
the 2000-2001 academic year. In October 2000, CIS staff members were placed in the schools, 
and planning and training were done quickly to accommodate an aggressive implementation time- 
table. Building an understanding of how CIS was expected to function in the context of Project 
GRAD was an important part of the early focus on strengthening implementation. Project GRAD 
Atlanta’s first Walk for Success occurred at Booker T. Washington High School in January 2001 . 

Beginning in the 2000-2001 academic year, the Project GRAD scholarship program be- 
came available to all entering ninth-grade students in the high school, and a scholarship coordi- 
nator was h ir ed to operate from an office there. 14 During the summer of 2001, the site had its 



13 One issue associated with this feeder pattern is that, traditionally, 50 percent of the students who attend 
Sylvan Hills Middle School go on to attend Terrell High School instead of Booker T. Washington High 
School. 

14 The Project GRAD scholarship is the second major scholarship offer available to students in Atlanta. 
The Georgia Lottery for Education funds the HOPE (Helping Outstanding Pupils Educationally) Scholarship 
Program. If high school students in Georgia graduate with an academic average of B or higher (a grade point 
average of at least 3.0), they are eligible for this scholarship, which may be used at public technical colleges 
and eligible public or private colleges and universities in Georgia. The scholarship pays tuition and mandatory 
fees not covered by other federal grants, and recipients also get a book allowance of up to $100 per quarter. 

(continued) 
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first summer institute, and over 130 students from Booker T. Washington High School attended 
the session for four weeks at Clark Atlanta University. Since its inception, Project GRAD At- 
lanta has placed considerable emphasis on building strong ties with the local institutions of 
higher education and on building partnerships with business and the community. 

Consistency Management & Cooperative Discipline SM (CMCD SM ) was introduced at 
the high school through an awareness training workshop held in the fall of 2001 and through 
various training sessions offered during the school year. The site had initially hoped to begin its 
overall implementation of Project GRAD with CMCD SM , but the implementation schedules 
across all the initiative’s sites could not accommodate that. Atlanta’ s implementation began dur- 
ing the 2002-2003 school year and drew some initial feedback that some of the classroom man- 
agement strategies were better suited for students in earlier grades. 

Project GRAD Atlanta has worked diligently to develop a strong presence within the 
community, and its ties to higher education have helped position the initiative in a high school 
that has many other programs and services in place and where it might otherwise have been less 
visible. 15 In 2004, Atlanta’s first class of Project GRAD scholars graduated 136 students — or 
23 percent of the senior class. 



Key Accomplishments of Implementation 

Although implementation of the high school program was not problem-free, Project 
GRAD can be credited with several important accomplishments that helped undergird the im- 
plementation efforts in Houston, Columbus, and Atlanta: 

• Across the Project GRAD high schools in the study, substantial numbers of 
principals and teachers were systematically trained to implement the pro- 
gram’ s classroom management component and, as a result, have learned new 
skills and an approach that — in most cases — has began to permeate other 
school activities and assemblies outside the classroom. 

• Project GRAD’s focus on providing in-school social service assistance and 
other supports to youth who need them has given these high schools a more 
accessible and more expedient mechanism to address students’ issues and 
problems that otherwise might interfere with the learning process. A variety 
of activities and strategies are being employed to connect with students and 



The scholarship is renewable each year, if the student maintains a cumulative grade point average of 3.0 for 
coursework attempted. Project GRAD Atlanta scholars may be eligible for both scholarships. 

15 In the 2002-2003 school year. Project GRAD Atlanta expanded to include the Carver High School and 
South Atlanta High School feeder patterns. 
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their families. Also, through such activities as the Walk for Success, Project 
GRAD has devised meaningful ways to engage parents and community 
members in school improvement efforts. 

• Project GRAD has helped these high schools focus on student achievement 
as a critical marker of their progress. Program staff have placed a premium 
on data analysis — to understand and track students’ progress toward meet- 
ing graduation requirements and to improve grade point averages, achieve- 
ment levels, and other student outcomes. Most important, students have 
gained an awareness of college requirements and have received a variety of 
support activities to help them gain access. 

• Project GRAD has been able to coexist with other education strategies and 
reforms within the same high school building, demonstrating that the initia- 
tive has the flexibility needed to complement a variety of approaches. The 
implementation experiences have shown that schools do not have to undergo 
a significant realignment of staff, students, or other programs in order to be- 
come a Project GRAD high school. 

• Project GRAD's scholarship offer and the emphasis on college-going at the 
critical ninth-grade transition point have helped students believe that college 
is within their reach and have helped them understand the pathway to get 
there. Each year, a growing number of students sign a Project GRAD con- 
tract, attesting to their commitment to meet the scholarship criteria. 

• Having met the scholarship requirements, Project GRAD scholars have now 
graduated from each of the high schools in the study, and many have gone on 
to college — one of Project GRAD’s highest-priority goals. 



A Framework for the Impact Analysis 

Chapters 3 and 4 discuss the impacts of Project GRAD on a set of key student out- 
comes, first in Houston and then in Columbus and Atlanta. The preceding discussion of the im- 
plementation of Project GRAD across those sites provides the context in which one must under- 
stand the impact analysis. The following observations about the implementation of Project 
GRAD provide a critical framework for that discussion and for the conclusions and implications 
explored in Chapter 5. 
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The nature of the differential impact test varies across the sites in this study. 



The impact analysis seeks to isolate the effects of Project GRAD at the high school level 
over and above the effects of other efforts and reform models that were used to improve student 
outcomes at the comparison schools in the study’s sites. To do this, the analysis includes trends in 
student outcomes at the Project GRAD high schools and trends at similar comparison high 
schools, recognizing that many of these schools were also using specific reform models. The pe- 
riod of this evaluation was a time of many different efforts to improve students’ academic out- 
comes, including efforts targeted specifically to the high school level. Thus, “business as usual” in 
the districts in this study was not an absence of reform activity or a static school improvement 
landscape; typically, many changes and other school improvement efforts were under way. 

Both the comparison schools and the Project GRAD schools received support for im- 
provement from the district level, partly in response to state and (recently) federal pressures. All 
the high schools and districts in the impact analysis faced academic challenges to improve, which 
drove their interest in Project GRAD as a reform initiative. But it appears that the district-level 
push for improvement played out somewhat differently across the sites and over time. Thus, in 
each site, student outcomes at the comparison schools are crucial to the analysis because they rep- 
resent the likely trends in the absence of Project GRAD. 

Further, the Project GRAD high schools varied somewhat in the combination of efforts 
that were in place. In Columbus, the implementation of Project GRAD occurred simultaneously 
with a major revamping of Linden-McKinley High School, and the initiative served as one of 
several levers for change. In Atlanta, Project GRAD joined several other reform efforts already 
in place at Booker T. Washington High School, but the initiative benefited from the district’s 
strong support and positioning of Project GRAD as an important element in its school im- 
provement plan. In Houston — partly because implementation began earlier — there were 
fewer existing high school reform initiatives, though some other, complementary efforts were 
present at Jefferson Davis High School. 

It is also important to remember that the strength of Project GRAD’s implementation var- 
ied across the sites. During the period covered in this analysis, the program's implementation in 
Houston was generally strong (acknowledging the difficulties of working in large urban school 
districts and low-performing schools). The expansion high schools in Houston benefited from the 
close oversight of the initiative’s founder, from their familiarity with Project GRAD as it was de- 
veloping at Davis High School, and from the proximity of the model's component developers. 

• Results must be interpreted in light of each site’s stage of program im- 
plementation. 
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By presenting the impact analysis at this point in the initiative’s timetable, this report is 
depicting Project GRAD’s effects at the high school level partway through an effort to put the 
program fully into place in some of the expansion sites: The model’s components were coming 
on-line in a phased-in approach and being strengthened, and students were working their way 
through elementary schools and middle schools that feed into the Project GRAD high schools. 
By the start of the follow-up period, this analysis begins to look at high schools as Project 
GRAD rolls out its components. This means that not all components are in place at the begin- 
ning of any follow-up period, and although all the components may technically be in place by 
the end of the follow-up period at the expansion high schools, they have not yet had much time 
to work together as a combined initiative, are still being refined and strengthened, and may not 
yet have developed their full momentum. This fact is important in interpreting the findings, but 
it does not suggest that analysis should be delayed until the program is fully in place in all the 
sites. Project GRAD’s gradual rollout of components and its focus on improvements in the early 
grades (to position students to take advantage of the college scholarships) contrast with other 
school reforms that more directly tackle the educational problems in high schools and more 
quickly push reforms into grades 9 through 12. The Project GRAD approach is based on an ar- 
gument that such an investment of effort over time will provide substantial payoffs in the long 
run. So the findings here — especially related to sites other than Davis High School — should 
be seen as part of an effort to track the early stages of this theory of action and as early results. 

• Feeder patterns vary in the extent to which they are “self-contained,” 
which complicates the challenges for Project GRAD. 

This analysis treats Project GRAD as an intervention that operates at the level of 
schools and feeder patterns, and it tracks the initiative’s effects on students’ academic perform- 
ance over time. The Project GRAD theory of action presumes that feeder patterns are suffi- 
ciently self-contained so that improvements at the elementary schools feeding into middle and 
high schools can change the educational outcomes at those schools. Yet two factors at the high 
school level in urban school districts can undermine this design assumption: student mobility 
and school-choice options. As an illustration, among Newark’s first-graders in the early 1990s 
who attended schools that later became Project GRAD schools, only 10 percent to 15 percent 
were attending the program’s designated high school in the ninth grade, and only about 50 per- 
cent to 60 percent of ninth-graders in this high school had attended a “Project GRAD” elemen- 
tary or middle school. The Columbus school district — which provides many school-choice 
options at each level (elementary, middle, and high school) — presents a similar challenge for 
Project GRAD’s implementation. 

To the extent that feeder patterns are not self-contained, many students leave Project GRAD 
elementary schools prior to the transition to Project GRAD secondary schools, and significant num- 
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bers of students in those secondary schools do not come from Project GRAD elementary schools. 
Depending on the extent of such factors, the reform faces challenges in implementation. 

Thus, the experience in Houston provides the clearest opportunity to examine Project 
GRAD at the high school level, for several reasons: the longest period of implementation, the 
“cleanest” system of assigning students to feeder patterns, and, in general — because of the 
program’s earlier start — fewer competing high school reform efforts. Although Atlanta and 
Columbus provide important opportunities to examine early experiences with Project GRAD, it 
is clear that those programs are at an earlier stage of development, which must be reflected in 
the types of student outcomes that are chosen for the analysis. 
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Chapter 3 



The Impacts of Project GRAD 
on High School Outcomes in Houston 

The typical urban school district has many reforms in place. Comprehensive school re- 
form models (which intend to change entire schools) have recently been the most common re- 
sponse to concerns about student achievement in urban schools and school systems. 1 School 
districts also often implement a variety of initiatives that are intended to improve teaching and 
learning across their entire school systems. As the discussion in Chapter 2 indicates, the districts 
in which Project Graduation Really Achieves Dreams (GRAD) were implemented were no ex- 
ception. During the period of implementation, most of these districts were attempting some kind 
of systemic reform. While such districtwide reforms were not always directed at the high school 
level, it was often the case that high schools with greater proportions of disadvantaged and mi- 
nority students were, in fact, participating in a variety of reform efforts. 

Therefore, the central concern for evaluators of Project GRAD — as is generally the 
case regarding any school reform effort — is really one of differential impacts. Reform efforts 
are not costless; they take financial, management, teaching, and political resources. Therefore, 
this analysis does not ask, “What is the effect of Project GRAD on student outcomes versus do- 
ing nothing?” Rather, this evaluation focuses on understanding whether Project GRAD leads to 
larger, similar, or smaller improvements in student outcomes than strategies that would have 
been in place if Project GRAD had not been implemented. In this analysis, the comparison is 
made between Project GRAD and the alternatives being tried in similar schools in the districts 
where Project GRAD was being implemented. 

Project GRAD’s ultimate goal is to increase high school graduation rates and college 
success among disadvantaged and minority students who might not otherwise succeed. Ideally, 
it would be best to measure Project GRAD’s effects directly on all these outcomes. For a variety 
of reasons (for example, the stage of implementation, limits of available data), it is not feasible 
to follow an appropriate sample of Project GRAD and comparison students for a long enough 
period of time to collect reliable data on longer-term outcomes related to college success. 2 
Therefore, this analysis focuses on effects on key steps along the way to college success: pro- 

Sce, for example, Bifulco, Duncombe, and Yinger (2005). 

2 In particular, in order to accurately assess the effects of Project GRAD on college success, a study would 
need to follow a sample of Project GRAD students and a sample of comparison students — most likely from 
similar high schools within the same district or districts — through the end of high school and several years 
after scheduled graduation, in order to assess postsecondary education outcomes among a representative sam- 
ple of program and comparison students. 
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motion or retention in the ninth grade, credits earned in high school, and — if enough time has 
elapsed — completion of a core academic college preparatory curriculum and graduation from 
high school. Box 3.1 lists the student outcomes examined in Chapters 3 and 4. 

For example, good attendance, on-time promotion from the ninth to the tenth grade and 
credits earned in particular classes are key predictors of on-time graduation. 3 In Houston — Pro- 
ject GRAD’s flagship site — and to a lesser extent in Atlanta and Columbus, program effects 
on these key predictors of high school success can be estimated. In Houston, where earlier im- 
plementation of the initiative provides a longer track record to evaluate, it is possible to estimate 
Project GRAD’s impact on high school graduation rates as well as on some indicators of readi- 
ness for college. 

The Project GRAD theory of action rests, in part, on the improved academic prepara- 
tion of students who reach high school. Even in Houston, where implementation began in the 
Jefferson Davis High School feeder pattern in 1993-1994, this aspect of the program is not yet 
fully in place. An evaluation at this point seems timely, however, given that the program has 
been in place for more than 10 years and that local policymakers and practitioners (for example, 
school superintendents) rarely have more than just a few years to evaluate academic improve- 
ment initiatives. Yet it is worth noting that this evaluation cannot rule out the hypothesis that 
findings would change over an even-longer follow-up period, giving Project GRAD high school 
students a chance to have received more of the intervention earlier in the feeder pattern. To gain 
insights into whether Project GRAD could alter the flow of better-prepared students into the 
ninth grade, the analysis, when possible, explores whether the feeder pattern into the Project 
GRAD high school is clearly defined. This may provide suggestive information on the longer- 
term improvements that could result from better-prepared students’ arriving at the Project 
GRAD high schools. 

In general, the analysis in this evaluation focuses on two questions: 

• How do key high school student outcomes change at Project GRAD high 
schools in the years following the program’ s implementation? 

• How do the changes in student outcomes at Project GRAD high schools 
compare with the changes at similar high schools from the same districts 
over the same time period? 

The next section of this chapter describes the analytic approach used to make these 
comparisons, illustrating how the analysis is designed to focus on the effects of Project 



3 See Kemple and Herlihy (2004) and Kemple, Herlihy, and Smith (2005). 
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Box 3.1 



Definitions of Key Student Outcomes 

Atlanta, Columbus, and Houston 

Attendance 

• Attendance rate: The total number of days during a school year that a student was 
marked as present divided by the total number of days that the student was listed as 
being enrolled. In Atlanta, the attendance rate was the number of days present divided 
by 180 days. 

• Regular attendance: Having an attendance rate of 90 percent or higher for the year. 

Enrollment Status 

• Promotion: An indicator of whether a student was in the next grade by the end of the 
following year - . By looking ahead one year, it is possible to determine whether a ninth- 
grade student was enrolled in the tenth grade by the end of that year or was repeating 
the ninth grade. 

Houston 

Course Credits Earned 

• Total credits earned: A cumulative total of all the credits that a student earned over 
the course of the first year of high school — all the credits earned in a student’s ninth- 
grade year. 

• Earned a credit in algebra: A designation, based on a student’s high school tran- 
script, indicating that the student completed and passed an algebra course. 

Tenth-Grade TAAS Scores 

• Pass rate: An indicator of whether a student passed the test, based on a passing score 
of 70 on the Texas Learning Index (TLI). This definition applies to both the math and 
the reading portions of the TAAS. 

• Took and passed the test the next year: An indicator of whether a ninth-grade stu- 
dent took and passed the test the following year (and was in the tenth grade on time). 

• Didn’t take the test on time: An indicator of whether a ninth-grade student took the 
test on time the following year. 

(continued) 
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Box 3.1 (continued) 



Graduation 



• Graduated on time: An indicator of whether a ninth-grade student graduated three 
years later from a school in the district. The data make it possible to follow a student 
over time and to observe whether the student graduated three year's after being en- 
rolled in ninth grade. 

• Ever graduated: An indicator of whether a student ever graduated from a school in 
the district, looking ahead at least three years. 

• Completed a core academic curriculum on time: An indicator of whether a student 
met three criteria: (1) graduating on time from a school in the district; (2) having a grade 
point average above 75 in the core academic subjects of math, English, social studies, 
and science; and (3) completing a core academic curriculum consisting of three math 
credits, four English credits, two social studies credits, and two science credits. 



GRAD at the high school level by addressing possible explanations for changes from sources 
other than Project GRAD. 4 

This analysis focuses on ninth-grade students as they enter high school beginning with the 
first year of Project GRAD implementation. In the first year of implementation, there was no op- 
portunity for a student to have been exposed to Project GRAD at the elementary or middle school 
levels. As the evaluation extends into the follow-up period, the expectation is that greater Project 
GRAD exposure through the feeder pattern would yield increasing impacts. 

The remainder of the chapter then presents findings from these comparisons in the 
Houston Independent School District (HISD), where Project GRAD model was first imple- 
mented and where there are currently five Project GRAD high schools. This analysis focuses on 
the first three high schools (Davis, Yates, and Wheatley) because the timing of Project GRAD’s 
implementation at these schools allows for somewhat longer experience with the program and 
more years of follow-up. This provides the fairest possible test of Project GRAD's effects. The 
analysis also presents the estimated effects for Davis High School itself. While it is difficult to 
draw statistical inferences from any one school, the longer-term patterns (10 years) at this flag- 
ship school may have implications for the Project GRAD model’s potential to effect change. 

4 As discussed later, these include other districtwide initiatives; changes in the composition of the student 
population; and the effects of local, state, or federal accountability systems. 
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Focusing on the three original Project GRAD high schools in Houston, this chapter ad- 
dresses the two questions posed above with respect to ninth-grade attendance and promotion rates, 
course-taking patterns, on-time graduation rates, and college readiness, as captured by on-time 
completion of a core academic curriculum. Chapter 4 focuses on Project GRAD's effects on early 
indicators of success at high schools in two of the early expansion sites, Columbus and Atlanta. 



Key Findings 

The analysis of the initiative’s effects at the first three Project GRAD high schools in 
Houston suggests several conclusions. 

• Student mobility and enrollment patterns meant that feeder patterns 
were not as airtight as initially expected. Even after 10 years of program 
implementation in the Davis High School feeder pattern, the average 
pre-high school exposure to Project GRAD was approximately 4.5 years. 

• Focusing on early indicators of high school success reveals that trends 
were generally upward and similar at the Project GRAD and compari- 
son high schools and that any differences in the extent of improvement 
are not statistically significant. 

• Project GRAD does not appear to have affected the percentage of ninth- 
graders who later graduated from an HISD high school. At both the 
Project GRAD and the comparison schools, this graduation rate slowly 
improved over the period of the study, and any differences in the extent 
of improvement are not statistically significant. 

• At Davis High School, increases in the percentage of ninth-graders who 
completed a core academic curriculum on time outpaced increases at the 
comparison schools, though this effect did not hold up across the three 
original Project GRAD Houston high schools as a whole. 



The Analytic Approach 

This analysis relies on individual student records obtained from each school district in 
this study. Staff in the three districts (in Houston, Columbus, and Atlanta) provided data on 
every student in every school, from several years before Project GRAD was implemented in 
each district through the spring of 2004. The resulting database across the three districts encom- 
passes 21 high schools and includes information on individual students' performance on the 
assessments administered in each district as well as data regarding students’ race/ethnicity, en- 
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rollment status, and (sometimes) eligibility for free or reduced-price lunch and their status as an 
English language learner. Importantly, the database links students over time as well as to the 
particular schools they attended. 

The most challenging aspect of evaluating Project GRAD or any similar reform is de- 
termining what would have happened in the absence of the program. In the literature of pro- 
gram evaluation, this is often referred to as the “counterfactual.” The most reliable technique for 
establishing the counterfactual is the random assignment of students or whole schools to a 
treatment group that has access to the program or to a control group that does not. However, in 
the case of Project GRAD, the feeder-pattern design of the intervention and the fact that imple- 
mentation had begun prior to the evaluation makes random assignment impossible. 

Therefore, an alternative approach is needed in order to estimate the outcome levels that 
what would have been observed in the absence of the program. To accomplish this goal, the 
analysis in this report applies an “interrupted time series” (ITS) approach with comparison 
groups . 5 This approach relies on the pre-Project GRAD history of student outcomes at the Pro- 
ject GRAD schools as the first stage in predicting the performance levels that would have been 
observed in the absence of the program. The basic logic of the approach is that — absent any 
special educational intervention or change in the school’s student body — the best predictor of 
future educational outcomes in a given school is the history of student outcomes in that same 
school. The approach therefore rests on two comparisons, discussed below. 

Comparison 1 — Deviation from the Baseline Average: The difference be- 
tween baseline student outcome patterns at Project GRAD schools and actual 
student outcome patterns in the years following program implementation 

While improvement over baseline patterns in a set of program schools is a first step in 
the analysis, it does not necessarily represent a “Project GRAD” effect, because there could be 
other explanations for this improvement. In particular, an important possible competing expla- 
nation for an observed change could be local events other than Project GRAD that affect stu- 
dent outcomes. If, during the same period as the particular program under study, there are major 
changes in the local context that are unrelated to the program (such as district-level initiatives, 
new state standards, or changes in curriculum), these changes could drive — in part or even 
completely — the improvements at the Project GRAD schools. Looking only at the changes in 
student outcomes at the Project GRAD schools does not make it possible to determine how 
much of any observed change from baseline patterns results from Project GRAD and how much 
is generated by changes in other local circumstances. Since reform is the rule rather than the 



5 For more detail on this approach and the manner in which it was applied to the evaluation of Project 
GRAD, see Snipes, Holton, and Doolitde (2006) and Snipes (2003). 
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exception in urban school districts, it seems particularly important to account for this phenome- 
non in the evaluation of Project GRAD. 

The primary mechanism for addressing this issue is to add comparison schools from the 
same local context to the analysis. In particular, by comparing the deviations from baseline pat- 
terns at Project GRAD schools with the deviation from baseline patterns at a set of comparison 
schools from the same district, the analysis seeks to “net out” the effects of other local events 
that coincided with Project GRAD. This approach adds a second comparison to the analysis. 

Comparison 2 — Difference in the Deviation from the Baseline Average: 

The difference between the deviation from the baseline patterns at Project GRAD 

schools and the deviation from baseline patterns at a set of carefully selected 

comparison schools from the same district 6 

If the amount of improvement in student outcomes since program implementation (that 
is, the deviation from the baseline) at the program schools is different from the amount of im- 
provement at the comparison schools over the same time period, this is evidence that the pro- 
gram had an impact on student performance over and above the effects produced by other local 
events. If, on the other hand, the improvements in student outcomes relative to the baseline are 
similar at both sets of schools, this suggests that the program did not have an effect on student 
outcomes over and above the effects of whatever reforms were present at the comparison 
schools or in the district as a whole. 

Choosing Comparison Schools 

As discussed in Appendix C, comparison schools are included in the analysis in order to 
provide an estimate of the progress that would have occurred at the program schools without 
Project GRAD. Therefore, the goal was finding a set of schools from the same district that, in 
the absence of any intervention, would be expected to perform similarly to the Project GRAD 
schools. The approach to selecting comparison schools used here once again rests on the logic 
that the most accurate predictor of a school’ s future performance is its prior performance. With 
that in mind, schools’ prior achievement levels were an important criterion for choosing com- 
parisons schools. 

Even with histories of similar student outcomes, schools that serve very different stu- 
dent populations might be expected to evolve differently over time, particularly in response to 
local events or district policies. Moreover, findings are more credible when comparisons have 
what the evaluation literature calls “face validity,” so the approach seeks comparison schools 



6 For details see. Bloom (2003) and Snipes (2003). 
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that serve similar “types” of students and that also exhibit similar student outcomes as the Pro- 
ject GRAD schools. 

With these priorities in mind, for each Project GRAD school, the analysis identified a 
set of comparison schools from the same district that were similar in terms of: 

• Average performance on standardized achievement tests in the years imme- 
diately preceding program implementation 

• The percentages of students in key demographic groups 7 

In most cases, this process resulted in several comparison schools for each Project 
GRAD high school. While there were minor variations across sites to account for data limita- 
tions and other circumstances, this approach was used to estimate program effects for the Pro- 
ject GRAD high schools in Houston as well as in Columbus and Atlanta (see Appendix C). 

Addressing Changes Over Time in Students’ Characteristics 

The analysis also addresses another possible competing explanation for changes in stu- 
dent outcomes: changes in the characteristics of students who are served by each school. Stu- 
dents’ characteristics at entry into high school are a predictor of success, so changes over time 
in their characteristics could produce changes in average student outcomes. In order to reduce 
any distortion created by such changes, the estimates of program effects control statistically for 
variation over time in such student characteristics as overage for grade and race/ethnicity. 8 The 
most noticeable change in student characteristics occurred at Davis High School, where the per- 
centage of students overage for ninth grade declined from about 75 percent at the baseline to 
about 50 percent by the end of the follow-up period. At Davis’ s comparison schools, the decline 
was from about 70 percent to 60 percent. 

7 ln Houston, the comparison high schools were limited to (1) those at which the average scores on the 
Texas Assessment of Academic Skills (TAAS) in the years immediately preceding Project GRAD's imple- 
mentation were within 0.25 standard deviation of the Project GRAD school in question and (2) those at which 
the percentage of students in the modal racial/ethnic group at the Project GRAD school was within 20 percent- 
age points of the percentage of students in the same group at the comparison schools. For example, if black 
students made up the largest racial/ethnic group at the Project GRAD school, then comparison schools were 
limited to those at which the proportion of black students was within 20 percentage points of the proportion at 
the Project GRAD school. 

8 In the Houston study, data limitations make it impossible to adjust the analysis to account for the percent- 
age of students who were economically disadvantaged. Usually, this is measured by students’ eligibility for 
free or reduced-price lunch. However, over the period of the analysis, the percentage of students for whom 
these data were missing changed substantially (from nearly 100 percent in some early years to under 5 percent 
in later years), making it impossible to determine the meaning of the percentage of students reported eligible. It 
is impossible to know how this might affect the results, but the available evidence suggests that if there is a 
bias, it is in favor of finding effects for Project GRAD. 
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The remainder of this chapter presents the results of the analysis of Project GRAD’s ef- 
fects on high school outcomes in Houston. Chapter 4 examines the effects at high schools in 
Columbus and Atlanta. 



Project GRAD’s Effects in Houston 

This section first describes the Project GRAD and comparison schools used in the 
analysis and then examines students’ enrollment in Project GRAD elementary and middle 
schools before reaching high school — a measure of prior exposure to the initiative’s educa- 
tional services and an element of its theory of action. The section next examines the impact of 
Project GRAD solely at Davis High School (following most outcomes through 10 years after 
the program was implemented) and then across the three Project GRAD high schools in Hous- 
ton, for which six years of implementation and follow-up data are available. 

As discussed in Chapter 2, the initial framework of the Project GRAD effort actually 
began to emerge at Jefferson Davis High School in 1988. However, the implementation of the 
model’s components as they now exist began in earnest in the 1994-1995 school year. Project 
GRAD was implemented next at Jack Yates High School in the 1996-1997 school year and then 
at Phillis Wheatley High School in the 1997-1998 school year. 9 The analysis that follows fo- 
cuses on understanding the effects of Project GRAD at these three high schools through the 
spring of 2004. Focusing on average effects at these three schools as a group provides a total of 
seven years of follow-up data across all three high schools. 10 

Baseline Characteristics at Project GRAD and Comparison Schools 

The student populations at the high schools served by Project GRAD Houston and the 
patterns of student outcomes in the year immediately preceding the program’s implementation 
can be characterized as follows: 

• Davis High School served a slightly higher percentage of Hispanic students 
than the comparison schools chosen for this analysis, but Davis and the com- 
parison schools exhibited similar levels of academic performance in the years 
immediately preceding Project GRAD's implementation. 



9 See Holland (2005). 

10 In order to avoid the biases that could result from a sample of schools that changed over time, the aver- 
age effects focus on the follow-up years for which data are available across all three high schools in the analy- 
sis. So, although Davis and Yates each have more than seven years of follow-up data for analysis, these addi- 
tional years are not included in the estimates of average effects. Instead, the analysis includes only the first 
seven years of implementation at each Project GRAD high school. 
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• The first three Project GRAD high schools in Houston and their comparison 
counterparts served similar students and exhibited similar baseline student 
outcomes. 

Table 3.1 compares the characteristics of students at Davis High School with the char- 
acteristics of students at the comparison schools and in the district as a whole during the years 
immediately preceding Project GRAD's implementation, that is, during the “baseline” period. 11 
The table indicates that though Davis High School was smaller than its comparison counter- 
parts, Davis and its comparison schools served similar student populations. For example, the 
vast majority of students at Davis were Hispanic, as was the case for the comparison schools 
chosen for this analysis. Davis and its comparison schools also served similar proportions of 
special education students and students who were classified as English language learners. 12 Im- 
portantly, the students at Davis and at the comparison schools also demonstrated similar out- 
comes. The attendance rate at both sets of high schools was 74 percent. At Davis, 43 percent of 
ninth-graders were promoted to the tenth grade each year, compared with 44 percent at the 
comparison schools. Moreover, the schools demonstrated similar tenth-grade pass rates on the 
Texas Assessment of Academic Skills (TAAS): 49 percent versus 51 percent passed the lan- 
guage arts test, and 38 percent versus 34 percent passed the math test. 

Table 3.2 reports average baseline student characteristics at Davis, Yates, and Wheatley 
High Schools and at their comparison schools. On average, the project GRAD and comparison high 
schools served similar students and demonstrated similar levels of baseline achievement, although 
again the Project GRAD high schools were smaller. Project GRAD schools served slighdy larger 
proportions of black students, similar proportions of Hispanic students, and fewer white students. All 
the schools had nearly identical attendance and promotion rates. Moreover, tenth-grade pass rates on 
the TAAS in both reading and math were within 3 percentage points of each other. 

Though not shown in the tables, it should be noted that Davis, Yates, and Wheatley 
High Schools served different racial/ethnic populations. The majority of students at Davis were 
Hispanic, whereas the majority at Yates were black. Wheatley was more evenly split: 58 per- 
cent of students were black, and 40 percent were Hispanic. 



"in general, the baseline period is defined as the three school years prior to the first year of program im- 
plementation. Because the initial year differed for each Project GRAD high school, the baseline period for each 
school includes different academic years. For Davis, inasmuch as data for some outcomes were not available 
prior to the 1992-1993 school year, the baseline period was sometimes defined as the two years prior to the 
1994-1995 school year. Other deviations from three years of baseline are noted. 

12 That is, students who were classified by the districts as “English Speaker of Other Language (ESOL).” 



34 




The Project GRAD Evaluation 
Table 3.1 

Baseline Characteristics of Students at Davis High School and Comparison Schools, 

1991-1992 Through 1993-1994 



Characteristic 


Davis High 
School 


Comparison 

Schools 


All Houston 
High Schools 


Average school size 








9th grade 


730 


1186 


716 


10th grade 


428 


697 


424 


11th grade 


311 


530 


337 


12th grade 


264 


481 


317 


Total 


1,733 


2,894 


1,794 


Race/ethnicity (%) 








Black 


12.8 


13.3 


44.7 


White 


1.2 


10.7 


14.5 


Hispanic 


84.1 


72.7 


36.5 


Asian 


1.9 


3.3 


4.2 


Other 


0.0 


0.0 


0.1 


Gender (%) 








Male 


48.9 


52.8 


50.3 


Female 


51.1 


47.2 


49.7 


Classified as English speaker of other language (ESOL) (%) 


51.0 


45.8 


34.2 


Classified for special education (%) 


5.1 


5.5 


5.9 


Characteristics of 9th-srade students 








Overage for grade 3 (%) 


77.2 


70.8 


61.1 


Attendance rate b (%) 


73.8 


73.7 


77.2 


Attendance rate greater than or equal to 90 percent 


66.3 


61.1 


69.0 


Promoted to the 10th grade 


43.2 


43.6 


50.8 


TAAS lOth-grade test scores 








Reading total 








Texas Learning lndex c (TLI) score 


63.3 


64.2 


68.9 


Pass rate 


49.1 


51.3 


64.7 


Math total 








Texas Learning lndex c (TLI) score 


60.8 


57.7 


61.5 


Pass rate 


37.9 


33.9 


44.0 


Total number of schools 


1 


4 


26 


(continued) 
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Table 3.1 (continued) 



SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

A cluster consists of a Project GRAD school matched with 4 comparison schools. Results in the Comparison 
Schools column reflect averages across these groups of non-Project GRAD schools. 

Baseline years consist of three years prior to implementation at Davis High School in 1994-1995. 
a A student is defined as overage for grade if he or she turns 15 before the start of the 9th grade. 
b Attendance rate is calculated by dividing the number of days present by the number of days enrolled. 

''The TLI is a continuous score that describes a student’s performance on the TAAS. The TLI is provided for 
both the TAAS reading and mathematics tests. The raw score on the TAAS is simply the number answered 
correctly on the test, and since this raw score can be interpreted only in relation to the number of items on the test, 
the score is limited in use. The TLI makes it possible to relate student performance to a passing standard and to 
compare student performance from year to year. In each year, the raw scores are standardized into TLI scores 
relative to the state's passing standard of 70. For TLI frequency distributions for each grade and subject, see Texas 
Education Center (2005). 



Exposure to Project GRAD in Elementary and Middle School 

A core element of Project GRAD' s design is that, rather than focusing on only one 
school at a time, the initiative attempts to implement its reforms in a feeder pattern of schools. 
Project GRAD identifies a set of elementary and middle schools that “feed” into a particular 
high school, and it implements a set of reforms that are intended not only to target that high 
school but also to improve the academic preparation of students before they arrive at that high 
school. Understanding the extent of students’ exposure to Project GRAD prior to their enroll- 
ment in high school can shed light on the extent to which the feeder patterns have functioned as 
the developers envisioned. The analysis suggests that: 

• The average amount of time that entering Project GRAD ninth-graders had 
spent in Project GRAD elementary and middle schools increased over the 
first several years of the program’s implementation. 

• In the last year of the follow-up period, on average, entering ninth-graders 
were receiving roughly half their elementary and middle school education in 
Project GRAD schools. 

The top panel of Figure 3.1 illustrates the average amount of prior exposure to Project 
GRAD among successive cohorts of entering ninth-graders at Jefferson Davis High School, the 
first Project GRAD site. The figure shows that ninth-graders entering Davis were exposed to 
Project GRAD prior to enrolling in high school, that their exposure substantially exceeded stu- 
dents’ exposure at the comparison schools, and that both the average amount of prior exposure 
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The Project GRAD Evaluation 
Table 3.2 



Baseline Characteristics of Students at the Project GRAD and 
Comparison High Schools in Houston 



Characteristic 


Project GRAD 
Schools 


Comparison 

Schools 


All Houston 
High Schools 


Average school size 


9th grade 


610 


903 


691 


10th grade 


307 


520 


399 


11th grade 


222 


379 


308 


12th grade 


194 


356 


299 


Total 


1,333 


2,158 


1,697 


Race/ethnicity (%) 


Black 


55.6 


43.8 


45.3 


White 


0.7 


10.0 


12.8 


Hispanic 


42.4 


42.3 


38.0 


Asian 


1.3 


4.0 


3.8 


Other 


0.1 


0.0 


0.1 


Gender (%) 


Male 


50.1 


49.8 


48.1 


Female 


49.9 


50.2 


51.9 


Classified as English speaker of other language (ESOL) (%) 


53.8 


53.9 


50.5 


Classified for special education (%) 


10.6 


8.7 


8.5 


Characteristics of 9th-srade students 


Overage for grade 3 (%) 


68.7 


64.1 


58.5 


Attendance rate b (%) 


74.4 


74.9 


74.6 


Attendance rate greater than or equal to 90 percent 


61.0 


62.1 


63.6 


Promoted to the 10th grade 


49.8 


49.6 


54.1 


TAAS lOth-grade test scores 
Reading total 


Texas Learning lndex c (TLI) score 


70.6 


71.1 


73.9 


Pass rate 


60.8 


63.3 


70.4 


Math total 


Texas Learning lndex c (TLI) score 


65.0 


63.3 


66.2 


Pass rate 


44.4 


41.8 


49.5 


Total number of schools 


3 


10 


84 



(continued) 
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Table 3.2 (continued) 



SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools column reflect averages across these groups of non-Project GRAD schools. 

Baseline years consist of three years prior to Project GRAD implementation. Project GRAD began at Davis 
High School in 1994-1995. Project GRAD began at Yates High School in 1996-1997. Project GRAD began at 
Wheatley High School in 1997-1998. 

a A student is defined as overage for grade if he or she turns 15 before the start of the 9th grade. 

b Attendance rate is calculated by dividing the number of days present by the number of days enrolled. 

''The TL1 is a continuous score that describes a student’s performance on the TAAS. The TLI is provided for 
both the TAAS reading and mathematics tests. The raw score on the TAAS is simply the number answered 
correctly on the test, and since this raw score can be interpreted only in relation to the number of items on the test, 
the score is limited in use. The TLI makes it possible to relate student performance to a passing standard and to 
compare student performance from year to year. In each year, the raw scores are standardized into TLI scores 
relative to the state's passing standard of 70. For TLI frequency distributions for each grade and subject, see Texas 
Education Center (2005). 



and the difference between Project GRAD and comparison schools increased over time. By 
2004 — Davis’s tenth follow-up year and the most recent year for which data are available — 
the average ninth-grader at Davis had spent approximately four and a half years in Project 
GRAD elementary or middle schools prior to starting high school, compared with an average of 
nearly zero years for ninth-graders at the comparison schools. For typical students, this repre- 
sents roughly half their education before high school. 

The bottom panel of Figure 3.1 illustrates students’ average exposure to the program 
across the first three Project GRAD high schools in Houston. By the sixth follow-up year, Pro- 
ject GRAD ninth-graders had roughly three years of exposure to the program in elementary and 
middle schools while ninth-grade students at the comparison schools had substantially less than 
a year of exposure. 

These graphs indicate that the feeder-pattern design succeeded in exposing some stu- 
dents to Project GRAD’s reforms prior to their enrollment in high school. However, the graphs 
also suggest that the feeder patterns in Houston were not “airtight” and that, on average, the Pro- 
ject GRAD students received a substantial portion of their elementary and middle school educa- 
tion outside the Project GRAD network. Part of Project GRAD’s strategy for improving high 
school outcomes was to improve students’ elementary and middle school education, leading to 
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Figure 3.1 (continued) 



SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Sample consists of students for whom administrative records exist between the 1990-1991 and 2003- 
2004 academic years. 

Project GRAD began at Davis High School in 1994-1995. Project GRAD began at Yates High School in 
1996-1997. Project GRAD began at Wheatley High School in 1997-1998. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 



better-prepared entering students. However, student mobility and enrollment patterns limit the 
extent to which this approach can produce change at the high school level. 



The Impacts of Project GRAD on Early High School Outcomes 

Project GRAD's ultimate aim is to help students graduate from high school and succeed 
in college. Previous research has shown that the ninth grade is a crucial transition period for stu- 
dents and that their chances of succeeding in high school and graduating on schedule are highly 
correlated with their attendance and performance in the ninth grade . 13 For example, passing alge- 
bra by the end of the ninth grade is a milestone in students’ academic progress and a prerequisite 
for taking many other courses necessary for graduation from high school and success in college. 
Moreover, whether or not students are promoted on time from the ninth grade to the tenth grade is 
a key predictor of whether or not they will graduate from high school in four years . 14 

The analysis presented below first examines Project GRAD’ s effects on precursors to 
graduation at the three Project GRAD high schools in Houston: Davis, Yates, and Wheatley. 
Key outcome measures include attendance, credits earned during the ninth grade, promotion 
rates, and students’ pass rates on the TAAS. 

Attendance 

• In general, in the years following Project GRAD’s implementation, ninth- 
grade attendance rates at Davis High School or across the initial three Project 
GRAD high schools as a whole did not improve. Average attendance rates 
across the comparison schools did increase during this time, suggesting that, 
on average, Project GRAD had a negative affect on ninth-grade attendance. 

13 See Kemple and Herlihy (2004) and Kemple, Herlihy, and Smith (2005). 

14 See Roderick ( 1993). 
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Figure 3.2 presents the estimated impacts of Project GRAD on attendance at Davis 
High School and at Davis, Yates, and Wheatley High Schools combined. The top panel of the 
figure presents the estimated effects of Project GRAD on the percentage of ninth-grade students 
at Davis who had attendance rates greater than or equal to 90 percent. Below the graph, the fig- 
ure reports the baseline average percentages of students who met this threshold at Davis (in the 
top panel) and at Davis, Yates, and Wheatley (in the bottom panel) and at the comparison 
schools. 



The bars in the graph represent the difference between these baseline averages and the 
average percentage of students who had attendance rates of 90 percent or greater. That is, the 
bars show the “deviation from the baseline average” in each of three periods: the first four years 
after implementation (Years 1-4), the next three years of implementation (Years 5-7), and the 
last three years of implementation (Years 8-10). The shaded bars represent the deviation from 
the baseline average at the Project GRAD school(s), and the unshaded bars represent the devia- 
tion from the baseline average at the comparison schools. Bars falling above the zero line repre- 
sent increases in student outcomes after the baseline period, whereas bars falling below the zero 
line represent declines in outcomes relative to the average before Project GRAD’s implementa- 
tion. The figure also reports the difference between the deviation from the baseline at the Project 
GRAD schools and the deviation from the baseline at the comparison schools. This is the esti- 
mated program “impact,” that is, the estimated effect of Project GRAD over and above the 
changes that would have been observed without the program. 

Figure 3.2 shows that, in the baseline years, Davis High School had somewhat higher 
levels of attendance than its comparison schools — approximately 70 percent versus 63 percent. 
After falling during the first four years of implementation, the percentage of students who had 
“high” attendance returned to 7 1 percent by the final three follow-up years. On the other hand, 
the percentage of students who had 90 percent attendance at the comparison schools increased 
from 63 percent to 7 1 percent over the course of the entire follow-up period. While these differ- 
ences are not statistically significant (that is, one cannot reject the possibility that they arose by 
chance), they suggest a negative relationship between Project GRAD and student attendance at 
Davis High School. 15 

The bottom panel of Figure 3.2 presents average impacts of Project GRAD on atten- 
dance at Davis, Yates, and Wheatley High Schools combined. This panel presents average pro- 
gram effects separately for the first four years of Project GRAD’s implementation and for the 



15 ln particular, the analysis cannot establish with greater than 90 percent confidence that this difference is 
due to systematic differences in student outcomes rather than to random fluctuations in the data. 
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Figure 3.2 (continued) 

SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline 
average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 



next three years of available follow-up data. 16 The evidence here suggests a similar story, and 
these differences are statistically significant. 

As the bottom panel of the figure indicates, in the baseline years, an average of 60 per- 
cent of ninth-graders at the three Project GRAD high schools had attendance rates of 90 percent 
or higher, similar to the 63 percent of ninth-graders at the comparison schools. The impact esti- 
mates presented in Figure 3.2 suggest that these attendance rates moved apart in the years fol- 
lowing Project GRAD's implementation. In particular, during Years 1-4 of follow-up, the per- 
centage of Project GRAD ninth-graders who had high attendance dropped to about 53 percent, 
while the percentage of comparison school ninth-graders who had high attendance increased to 
about 65 percent. Comparing the drop of slightly more than 6 percentage points at the Project 
GRAD schools with the increase of just under 2 percentage points at the comparison schools 
suggests a net impact of negative 8 percentage points, which is statistically significant. The pat- 
tern appears to have persisted through Years 5-7 of follow-up. 



16 Note that follow-up years are defined relative to the implementation of Project GRAD at each high 
school and do not represent consistent calendar years. For example, the first year of implementation at Davis 
occurred two or three years before the first year of implementation at Yates and at Wheatley. Nevertheless, the 
impacts at each school in these different calendar years are combined in order to estimate the program’ s aver- 
age impact across the three high schools in the “first year of implementation.” 
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Credits Earned 

• Average credits earned during ninth grade and the percentage of students 
earning an algebra credit in ninth grade generally increased during the fol- 
low-up year at both Davis High School and the three Project GRAD Houston 
high schools combined. However, improvements at the comparison schools 
were similar. 

Algebra is an important gateway course for ninth-graders. Passing this course is typi- 
cally a requirement for high school graduation, and algebra is a prerequisite for other high 
school math courses that are required for admission to college. Figure 3.3 illustrates the esti- 
mated program impacts for the percentage of ninth-graders completing an algebra credit. As 
shown in the top panel of the figure, at Davis High School during the baseline year, 27 percent 
of ninth-graders earned an algebra credit. 17 In Years 2-4 of follow-up, this rate increased 16 per- 
centage points, reaching an average of 43 percent. Over the same period, the rate at the com- 
parison schools increased by 13 percentage points. The difference between these deviations 
from baseline, while slightly favoring Davis, is not statistically significant. 

Progress in algebra credits continued through Years 5-7 of the follow-up period, when 
the pass rates increased to 45 percent at Davis, representing a total increase of 17 percentage 
points over the baseline average. During the same period, the pass rate in algebra at the com- 
parison schools grew by 16 percentage points. The difference between these two patterns is 
small and is not statistically significant. 

Interestingly, the pass rates in algebra among ninth-graders at Davis High School and at 
its comparison schools dropped below the baseline level in Years 8-10 of the follow-up period. 
In particular, the pass rate at Davis fell to 19 percent, while the rate at the comparison schools 
fell to 16 percent. Conversations with personnel from the Project GRAD Houston staff suggest 
that the dropoff was simultaneous with the increased enrollment in algebra in middle school, 
which is not captured in the HISD high school student records. 



17 For the two baseline years just prior to the start of Project GRAD at Davis High School and in the first 
implementation year, data on course credits earned at Davis are missing from the HISD student records system. 
During this period, Davis High School staff maintained their own computerized records system, and these data 
were never included in the HISD system. This data problem is not present for the other Project GRAD high 
schools or for the comparison schools. In order to avoid calculating an artificially low baseline level at Davis 
for outcomes based on credit data, this analysis uses only one baseline year for Davis High School and starts 
the follow-up and impact calculation at Year 2. At all the comparison schools and the other two Project GRAD 
high schools, which do not have this data problem, the analysis uses three baseline years in calculating the 
baseline average, in order to take advantage of the available data. 
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Figure 3.3 (continued) 

SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline average 
and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are adjusted 
to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 
“Used one year of baseline data for Davis. 
b Years 1-4 for Davis includes only Years 2-4. 



The bottom panel of Figure 3.3 illustrates the average program effects on ninth-grade al- 
gebra credits across the three Project GRAD high schools in this analysis. It shows that the aver- 
age progress on ninth-grade algebra credits is smaller than at Davis High School alone and that the 
comparison schools made similar progress. In particular, across the three Project GRAD high 
schools in this analysis, algebra pass rates increased from an average of 20 percent during the 
baseline period to 31 percent at the end of follow-up, while the comparison schools’ pass rate in- 
creased from 19 percent to 30 percent between the baseline period and the end of follow-up. 

Figure 3.4 illustrates the estimated program effects on the total number of academic 
credits that ninth-graders earned. As is the case for the algebra credits, the top panel of the fig- 
ure shows that there was progress over the course of the follow-up period at Davis High School. 
The progress appears to have been concentrated in the second follow-up period (Years 5-7), 
after four years of Project GRAD’s implementation. The observed improvements were similar 
at the comparison schools. The bottom panel for the first three Project GRAD high schools 
combined shows almost no difference between the improvements at Project GRAD and the 
comparison schools. 

Promotion from Ninth to Tenth Grade 

• During the baseline period, an average of less than 50 percent of ninth- 
graders at the Project GRAD and comparison schools were promoted to the 
tenth grade each school year. 
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Figure 3.4 (continued) 



SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline average 
and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are adjusted 
to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 
a Used one year of baseline data for Davis. 
b Years 1-4 for Davis includes only Years 2-4. 



• In the years following Project GRAD's implementation, Davis High School 
showed modest improvements in promotion rates. Though these improvements 
appear to exceed the improvements at the comparison schools during the later 
part of the follow-up period, the differences are not statistically significant. 

• Across the three original Project GRAD Houston high schools as a group, the 
progress over time was more modest, and the average differences between 
the changes at the Project GRAD and at the comparison schools are smaller 
than the differences at Davis, suggesting a lack of effects on promotion rates 
at the other two high schools. 

Figure 3.5 reports average ninth- to tenth-grade promotion rates during the baseline pe- 
riod and illustrates the progress over time and the estimated program effects at Davis High 
School (see the top panel). In the years preceding Project GRAD’s implementation, an average 
of approximately 45 percent of ninth-graders at both Davis and its comparison schools were 
promoted to the tenth grade. 18 The extent of progress made on this measure appears to have 
fluctuated over time. However, by the end of the follow-up period, Davis’ s promotion rate had 



s I n particular, the analysis focuses on the percentages of ninth-graders who — based on enrollment re- 
cords from HISD — were repotted as enrolled in the tenth grade by the end of the following school year. 
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Figure 3.5 (continued) 

SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline 
average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 



improved by 7 percentage points, compared with a net improvement of just 1 percentage point 
at the comparison schools. Although this suggests a positive impact of 6 percentage points at 
Davis, inasmuch as the difference is not statistically significant, one cannot rule out the possibil- 
ity that it is driven by chance. 

Like the outcomes for course credits, the ninth- to tenth-grade promotion rates shown in 
the bottom panel of the figure reflect less progress across the three Project GRAD feeder pat- 
terns as a whole, and the differences between the Project GRAD and comparison schools are 
smaller. This suggests that effects on promotion rates may be present at Davis but that they are 
not present at the other two Houston high schools in this analysis. 

Student Achievement 

• In the years following Project GRAD’s implementation, the percentage of 
ninth-graders at the Project GRAD schools who took and passed the tenth- 
grade reading and math portions of the TAAS in the following year increased. 

• The progress at the comparison schools was similar, and any small differ- 
ences in improvements are not statistically significant. 

In the spring of 2003, the State of Texas changed its testing regime, switching from the 
Texas Assessment of Academic Skills (TAAS) — which had been administered since 1994 — 
to the Texas Assessment of Knowledge and Skills (TAKS). Because the change in tests makes 
it difficult to compare results over time, the analysis of student achievement does not go beyond 
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the spring of 2002. This yields seven years of follow-up data for the Davis feeder pattern and 
four years of follow-up data for the three Project GRAD feeder patterns combined. 

Figures 3.6 and 3.7 report estimated program effects on the percentage of ninth-graders 
who took and passed the math and reading portions of the TAAS in the following year. 19 The 
figures show that, in general, the “take-and-pass” rates in reading and math at Project GRAD 
and comparison schools improved relative to the baseline period. With respect to math, the pat- 
terns for Davis and the three Project GRAD high schools combined are similar. For example, 
the rates at the Project GRAD high schools (in the bottom panel of Figure 3.6) averaged just 
under 13 percent during the baseline period and grew to about 21 percent during the first four 
years of follow-up. The net difference between student achievement during the baseline period 
and the follow-up period is just above 7 percentage points. On the other hand, over the same 
period of time, the rates at the comparison schools improved from 13 percent to 23 percent — a 
difference of 10 percentage points. This negative difference of 3 percentage points is not statis- 
tically significant. 

Figure 3.7 compares the progress over time and reports the estimated program effects 
on pass rates for the “take-and-pass” rates for the reading portion of the TAAS. In Years 1-4 of 
follow-up, average reading rates in the Project GRAD high schools improved from 17 percent 
to 23 percent (see the bottom panel of the figure). A similar pattern occurred at the comparison 
schools, suggesting that Project GRAD had no net effect — positive or negative — over and 
above the progress in reading that would have occurred in the absence of the program. 



The Impacts of Project GRAD on High School Graduation and 
Preparation for College 

Project GRAD' s ultimate goal is to increase the number of students who graduate from 
high school and succeed in college. Project GRAD Houston provides the best available evi- 
dence regarding the extent to which the initiative has had an impact on how many students 
reach this goal. In particular, Project GRAD Houston is the only site that has been in place long 
enough to track several years of program effects on four-year high school graduation rates. As 



19 This measure is used to avoid problems that might be caused by examining tenth-grade pass rates. A po- 
tential problem with the pass-rate measures has to do with differences in test-taking patterns between the Pro- 
ject GRAD and comparison schools. In particular, to the extent that any differences in promotion or test-taking 
patterns across schools result in different percentages or in different populations of students taking the tests, 
estimates based on pass rates among test-takers could distort the impact estimates. The analysis presented in the 
text is not subject to any bias favoring schools that — for whatever reason — test fewer ninth-graders the fol- 
lowing year or test a select subgroup of tenth-graders. Appendix B continues these findings by examining dif- 
ferences in test-taking patterns between the Project GRAD and comparison schools, by estimating impacts on 
the percentage of ninth-graders who took the tenth-grade test the following year. 
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Figure 3.6 (continued) 



SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline average 
and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are adjusted 
to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 



of 2004 — the end of this study’s follow-up period — Project GRAD had been in existence at 
Davis High School for ten years, at Yates High School for eight years, and at Wheatley High 
School for seven years. Given that, it is possible to follow the effects of Project GRAD on four- 
year graduation rates and other end-of-high school outcomes for several cohorts of ninth- 
graders. However, given the phase-in of components, not all components were in place each 
year that Project GRAD operated in the school. 

Four-Year Graduation Rates 

• In the years preceding Project GRAD's implementation in the HISD, ap- 
proximately 70 percent of students at both Project GRAD and comparison 
schools did not graduate from high school in the HISD within four years of 
enrolling in the ninth grade. 

• In the years following Project GRAD’s implementation, graduation out- 
comes at Davis High School modestly improved, but similar improvements 
took place at the comparison high schools chosen for this analysis. 

• Looking across the three original Project GRAD Houston high schools as a 
whole, neither the Project GRAD nor the comparison high schools showed 
substantial progress on average graduation rates. 
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The Project GRAD Evaluation 
Figure 3.7 
Impact Estimates: 

Percentage of Ninth-Graders in Houston Taking and Passing the Tenth-Grade 
Reading Test the Next Year, Follow-Up Results 
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Figure 3.7 (continued) 



SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline 
average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 



Most people think of “on-time” graduation from high school as graduating within four 
years of enrollment in the ninth grade. However, most school districts in the country do not pub- 
lish this information, focusing instead on the total number of graduates annually. These counts 
can be influenced by several factors, including changes in school size, student mobility, and 
enrollment patterns. As such, changes in the number of graduates may reflect factors other than 
educational improvements that increase the chances that students will progress through school. 
The counts are also affected by changes in the number of ninth-graders over time . 20 Some dis- 
tricts focus on the percentage of students in each twelfth-grade class who graduate at the end of 
that academic year. The concern about such measures is that much of the dropout problem oc- 
curs before the measures begin counting students, and so they substantially underestimate the 
percentage of students who do not finish high school. 

The analysis of graduation rates presented here focuses on successive cohorts of first- 
time ninth-graders who were enrolled at Project GRAD and comparison high schools in the 
spring of the academic year. It examines whether or not, according to the districts’ data, these 
students are identified as graduates in any HISD high school three years later (that is, the sched- 
uled end of the twelfth grade ). 21 Because students transfer in and out of school districts, and 



20 For example, the number of ninth-graders at Davis High School fluctuated from a low of about 600 to a 
high of about 900 over the period of this analysis. 

“'Previous research suggests that, because many student repeat the ninth grade, graduation rates for all 
ninth-graders can be difficult to interpret (see, for example, Miao and Haney, 2004). In particular, besides be- 
ing a function of events that occur in the school year in question, graduation rates based on an entire class of 
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from public to private schools, these numbers understate the proportions of ninth-graders who 
graduated from any high school within four years. In particular, they do not account for the per- 
centage of students who left the HISD and graduated from another district. Instead, these esti- 
mates reflect the percentage of first-time ninth-graders who graduated in a timely basis from 
anywhere within HISD. 22 

Looking across the three original Project GRAD high schools, the evaluation can follow 
the average effects of Project GRAD Houston on most student outcomes through seven succes- 
sive cohorts of students — the first seven classes of ninth-graders at the participating Project 
GRAD high schools. However, unlike measuring other outcomes, calculating graduation rates 
requires four years of follow-up data for each cohort of students. Therefore, for graduation and 
related outcomes, the combined findings across all three Project GRAD high schools cover only 
the first three classes of ninth-graders enrolled; for Davis High School alone, the analysis covers 
the first six classes of entering ninth-graders. 

Figure 3.8 presents the average distribution of graduation outcomes at the Project 
GRAD high schools in the baseline years immediately preceding the program’s implementa- 
tion. As the figure shows, the majority of entering ninth-graders did not graduate within four 
years. In fact, over 40 percent of the entering ninth-grade class were no longer enrolled in the 
district three years later. Of the ninth-graders in the sample, 42 percent left the district before 
their scheduled end of the twelfth grade; 17 percent remained enrolled in the district, but — as 
of three years later — had not reached the twelfth grade; and 1 1 percent made it to the twelfth 
grade but did not graduate on time. Even among those who reached the twelfth grade on time, 
about a quarter did not graduate on time. In the end, approximately a third (30 percent) of enter- 
ing ninth-graders graduated from HISD four years later. 23 



ninth-graders are also a function of events in previous academic years, including changes in rules and norms 
for promotion. The estimated effects of Project GRAD on successive cohorts of all ninth-graders do not sug- 
gest systematically different patterns of effects than the estimates presented in this report. These estimates are 
presented in Appendix Figure A.2. 

22 Unless there is a systematic relationship between Project GRAD and student mobility out of the district, 
this mobility would not undermine the validity of the comparisons between graduation rates at Project GRAD 
versus comparison high schools. 

23 The estimates in this report focus on “on-time” graduation, that is, graduation within four years of en- 
rollment in the ninth grade. This analysis also includes estimated effects on graduation, irrespective of whether 
that took longer than four years. These estimates are repotted in Appendix A. While they suggest slighdy 
higher graduation rates, they do not suggest impact patterns that differ from those summarized in this report. 
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The Project GRAD Evaluation 
Figure 3.8 

Houston: Distribution of First-Time Ninth-Graders Who Graduated On Time 
Three Years Later, Average Prior to Project GRAD 



Not promoted to 
12th grade 
17% 



Exited by 12th grade 

42% 




Graduated on time 
30% 

J 



Promoted to 12th 
grade but didn't 
graduate on time 
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SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: The baseline period for Davis High School is 1991-1992 to 1993-1994. The baseline period for Yates 
High School is 1993-1994 to 1995-1996. The baseline period for Wheatley High School is 1994-1995 to 1996- 
1997. 

Baseline averages are unadjusted. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 
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Figure 3.9 presents the estimated effects of Project GRAD on high school graduation at 
Davis High School and at the three original Project GRAD high schools as a whole. 24 The figure 
supports three important findings. First, the majority of ninth-grade students in any particular fol- 
low-up year did not graduate within the next four years. Second, on-time graduation rates at Davis 
improved modestly in the years following Project GRAD’s implementation, and the combined 
rates at the three Project GRAD high schools remained essentially the same. Third, though the 
picture at Davis is somewhat more encouraging, the patterns there and at Project GRAD high 
schools overall do not differ significantly from the patterns at the comparison schools. 25 

During the baseline period at Davis High School (in the top panel of Figure 3.9), an av- 
erage of 35 percent of Project GRAD ninth-graders graduated within the next four years, com- 
pared with about 30 percent of ninth-graders at the comparison schools. During the next three 
years, the graduation rate increased by about 2 percent at Davis and at the comparison schools 
chosen for the analysis. By Years 4-6 of follow-up, graduation rates at Davis had improved by 
about 7 percent over the baseline levels, compared with a change of about 5 percent at the com- 
parison schools. This 2 percent difference is not statistically significant. Across the three Project 
GRAD high schools as a whole (in the bottom panel of the figure), the four-year graduation rate 
remained virtually unchanged — increasing by less than 1 percentage point — whereas the rate 
increased by approximately 3 percent at the comparison schools. Again, this difference is not 
statistically significant. 

Completing a Core Academic Curriculum “On Time” 

The Project GRAD college scholarship offer is often referred to as the “cornerstone” of 
the intervention. It is used as a lever to increase high school students’ focus on going to college 
and also to improve their access to college once high school is over. To be eligible for the schol- 
arship, students must meet several criteria: They must take an academic core curriculum (in or- 
der to meet the requirement of successfully completing the courses designated as college pre- 
paratory by their state, district, and school); they must maintain a 2.5 grade point average; and 



24 The number reported in Figure 3.9 as the baseline average for all three Project GRAD schools (32.1 per- 
cent) differs slightly from the one presented in Figure 3.8 (30.0 percent) because it is adjusted to account for 
students' demographic characteristics. 

25 When estimates are generated without controlling for demographic characteristics (such as the percent- 
age of Flispanic students and the percentage of students who are overage for their grade), they suggest larger 
gains in graduation rates. They also suggest some differences between the growth at Davis High School and its 
comparison schools. The fact that these changes and the differences between the schools are mostly eliminated 
by controlling for students’ demographic characteristics suggests that demographic changes — rather than the 
effect of any reform per se — may have driven the observed changes in “unadjusted” graduation rates. In other 
words, the evidence here suggests that, rather than changing the graduation rate among all students, the imple- 
mentation of Project GRAD is coincident with changes in the student population being served, most notably a 
decline in the percentage of students who were overage for their grade. 
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The Project GRAD Evaluation 
Figure 3.9 
Estimated Impacts: 

Percentage of First-Time Ninth-Graders in Houston Who Graduated On Time 
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Figure 3.9 (continued) 

SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline 
average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 

This figure focuses on analysis of successive cohorts of first-time ninth-graders in Project GRAD and 
comparison schools and whether students graduated on time three years later. 



they must graduate from high school on time. Students who meet these milestones and who at- 
tend the two Project GRAD summer institutes and sign the commitment contract can receive a 
college scholarship of $1,000 to $1,500 per year. Given the importance of the scholarship, the 
extent to which students meet the eligibility requirements represents an important outcome for 
the program. The scholarship criteria also represent reasonable proxies for students’ preparation 
for college. As such, progress on this outcome speaks to Project GRAD's success in achieving 
its ultimate outcome: successfully preparing students for college. 

This section presents estimates of the effects of Project GRAD on a key element of the 
scholarship requirement: the proportion of first-time ninth-grade students who complete a core 
academic curriculum within four years of starting high school. 26 

• The vast majority of first-time ninth-grade students did not complete a core 
academic curriculum by the scheduled end of high school. 

• Across the three Project GRAD high schools, the percentages of students 
completing this increased over the course of the first few years after Project 



26 In particular, students are classified as having completed this if they received an average grade of 75 out 
of 100 in their core courses; if they earned four credits in English, three in math, two in science, and two in 
social studies; and if, according to HISD records, they graduated from high school within tour years of being 
enrolled in the ninth grade. 
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GRAD's implementation. However, ninth-graders at the comparison schools 
showed similar progress. 

• At Davis High School, the increases in the percentage of students completing 
this curriculum significantly outpaced the changes that occurred at the com- 
parison schools, supporting a conclusion that Project GRAD at Davis had a 
substantial positive effect. 

Figure 3.10 shows that, during the baseline period, the vast majority of first-time ninth- 
grade students at the three Project GRAD high schools in Houston did not complete a core aca- 
demic curriculum on time. Only 8 percent achieved this milestone. The majority (69 percent) 
did not graduate from high school within the next three years, while 23 percent were identified 
as graduating on time but did not meet the other criteria. 

Figure 3.11 shows the estimated effects of Project GRAD on the percentage of ninth- 
graders completed a core academic curriculum on time four years later. The top panel shows 
findings for Davis High School, and the bottom panel shows findings for the three Project 
GRAD high schools combined. 27 The analysis includes three cohorts of ninth-graders for Davis 
and for the Project GRAD high schools as a whole. 

The figure indicates that the percentage of students who completed a core academic 
curriculum on time increased substantially at both Davis and its comparison schools. It also in- 
dicates that the increases at Davis exceeded those at the comparison schools and that these dif- 
ferences are statistically significant. During the baseline period at Davis, 9 percent of first-time 
ninth-graders met this requirement. In the years following Project GRAD’s implementation, the 
percentage of ninth-graders meeting these criteria at Davis increased by 12 percentage points, to 
22 percent. This exceeds the progress at the comparison schools, which improved from 12 per- 
cent during the baseline period to 17 percent by the end of follow-up. The gain at Davis repre- 
sents a statistically significant positive effect of nearly 7 percentage points. 

While the previous section on four-year graduation rates suggests that Project GRAD 
did not have a significant positive effect at Davis High School, these results imply that it did, in 
fact, have significant positive effects on the course-taking and grades of Davis students, essen- 
tially increasing the extent to which the school’ s graduates were ready for college. 



27 Because data on credits earned at Davis High School are not available for several early years (see foot- 
note 16), the follow-up period available for this analysis is shortened. Specifically, at Davis the analysis uses a 
quasi-baseline period made up of two early implementation years. Because of this shift to a quasi-baseline, 
only three cohorts of ninth-graders can be included in the analysis. If Project GRAD had already started to im- 
prove academic performance of Davis students during the quasi-baseline period, this analytic choice creates a 
higher benchmark against which later cohorts are compared. This strengthens the finding of a positive effect on 
completion of a core academic curriculum on time at Davis High School. 
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The Project GRAD Evaluation 
Figure 3.10 

Houston: Distribution of First-Time Ninth-Graders Who Completed Core 
Academic Curriculum Requirements On Time Three Years Later, 
Average Prior to Project GRAD 
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SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: The quasi-baseline period used for Davis High School is 1995-1996 to 1996-1997. The baseline period 
for Yates High School is 1993-1994 to 1995-1996. The baseline period for Wheatley High School is 1994-1995 to 
1996-1997. 

Baseline averages are unadjusted. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 
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The Project GRAD Evaluation 
Figure 3.11 
Estimated Impacts: 

Percentage of Ninth-Graders in Houston Who Completed Core Academic Curriculum 

Requirements On Time 
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Figure 3.11 (continued) 

SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline 
average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 

This figure focuses on analysis of successive cohorts of first-time ninth-graders at Project GRAD and 
comparison schools and whether students met the core academic graduation requirements. 

a The Davis cluster used 1997-1998 as implementation Year 1. Two quasi-baseline years (1995-1996 and 
1996-1997) were used. 



The bottom panel of Figure 3.11 reports the estimated impact of the program on the 
percentage of first-time ninth-graders meeting the criteria at the three Project GRAD high 
schools as a whole. The average increases in the percentage of students meeting these criteria 
are more modest and are essentially identical to the changes at the comparison high schools. 
This suggests that the positive effects on this outcome at Davis High School did not occur at 
Yates and Wheatley High Schools. 
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Chapter 4 



The Impacts of Project GRAD on High School Outcomes 
in the Columbus and Atlanta Expansion Sites 

A key goal of this evaluation is to understand the effects of Project Graduation Really 
Achieves Dreams (GRAD) on high school students’ outcomes in two of the initiative’s early 
expansion sites: Columbus, Ohio; and Atlanta, Georgia. For a variety of reasons, these two sites 
provide the best opportunity to study the intervention outside the three original Project GRAD 
high schools in Houston (see Chapter 3). Although neither of the expansion sites has operated 
the program enough years to assess its effects on graduation rates or completion of a core aca- 
demic curriculum, the available data do support an early look at the effects on precursors to 
graduation. The patterns of these early indicators may provide insights into whether Project 
GRAD will ultimately affect high school graduation and college success. 

Project GRAD Columbus began implementation at Linden-McKinley High School in 
the 1999-2000 academic year, while Project GRAD Atlanta was implemented at Booker T. 
Washington High School in the 2000-2001 academic year. The following analysis explores the 
program’s effects on precursors to graduation during the first four years of implementation in 
Columbus and during the first three years of implementation in Atlanta. Unfortunately, data 
limitations at these two sites prevent analysis of the effects of Project GRAD on course-taking 
and on credits earned in key courses. 

Like the analysis of the program in Houston, the analysis here focuses on understanding 
progress over time at the Project GRAD schools in the expansion sites and on ascertaining the 
extent to which it differs from the degree of progress at similar comparison schools chosen from 
the same districts. Because of the small samples in Atlanta and Columbus (one high school per 
district), the analysis has less statistical power to detect impacts than is the case for Houston. 



Key Findings 

• Attendance and promotion rates at the Project GRAD high schools in 
Atlanta and Columbus improved in the years following Project GRAD’s 
implementation, 

• Improvements at the Project GRAD high schools frequently appeared to 
outpace the changes at the comparison schools in the same districts, 
though many of the differences in outcomes between Project GRAD and 
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comparison schools that were found in these two expansion sites are not 
statistically significant. 

• Though these findings must be interpreted with caution, they suggest 
that Project GRAD may have had a positive influence on the precursors 
to graduation in these two expansion sites. 



The Baseline Characteristics of Students 

• On the whole, the Project GRAD high schools in both Columbus and Atlanta 
appear to have been serving student populations that were similar to the popu- 
lations at the comparison high schools chosen from within these districts. 1 

In Atlanta and Columbus, comparison schools were chosen on the basis of ninth-grade 
students’ characteristics and achievement levels in the three years immediately preceding Pro- 
ject GRAD’s implementation (see Appendix C). Table 4.1 presents the average baseline charac- 
teristics of ninth-graders at the Project GRAD Columbus high school (Linden-McKinley High 
School) and the comparison schools chosen for this analysis. The estimates in the table suggest 
that the Project GRAD and comparison schools in Columbus served similar racial/ethnic popu- 
lations. For example, during the baseline period, 78 percent of the ninth-graders at both sets of 
high schools were African-American; the rest of the student body was made up largely of white 
students and of smaller numbers of Hispanic and Asian students. 

On the other hand, the students at the Project GRAD Columbus school were slightly 
more disadvantaged and demonstrated somewhat weaker academic outcomes than those at the 
comparison schools. For example, 52 percent of the students at the Project GRAD high school 
were economically or academically disadvantaged, compared with 39 percent of the students at 
the comparison schools. Perhaps more important, the Project GRAD students demonstrated 
lower pass rates on the state tests (35 percent versus 50 percent in reading comprehension; 14 
percent versus 18 percent in math). In addition, ninth- to tenth-grade promotion rates at the Pro- 
ject GRAD high schools were 49 percent, compared with 62 percent at the comparison schools. 

In Atlanta, in the years immediately preceding program implementation, the demo- 
graphic composition of students at the Project GRAD and comparison high schools were simi- 
lar. Table 4.2 compares the baseline characteristics of ninth-graders at the Project GRAD high 



'As was done for the Houston analysis, comparison schools in Columbus and Atlanta were chosen based 
on the similarity of students’ demographic characteristics and achievement outcomes. In particular, in both 
districts, comparison schools were limited to those that were within 20 percentage points of the modal ra- 
cial/ethnic group at the Project GRAD high school and those that were within 20 percentage points in terms of 
average baseline achievement (that is, pass rates on state achievement tests). 
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The Project GRAD Evaluation 
Table 4.1 



Baseline Characteristics of Schools and Students at the Project GRAD and 
Comparison High Schools in Columbus, 1996-1997 Through 1998-1999 





Project GRAD 


Comparison 


All Columbus 


Characteristic 


School 


Schools 


High Schools 


Average school size 








9th grade 


327 


317 


334 


10th grade 


196 


248 


237 


11th grade 


141 


205 


189 


12th grade 


94 


183 


162 


Total 


758 


953 


922 


Race/ethnicity (%) 








Black 


77.9 


78.4 


55.5 


White 


16.0 


17.6 


39.5 


Hispanic 


1.6 


0.6 


1.0 


Asian 


3.9 


1.9 


2.6 


Other 


0.0 


0.1 


0.2 


Gender (%) 








Male 


51.4 


49.6 


48.8 


Female 


48.6 


50.4 


51.2 


Classified as English speaker of other language (ESOL) (%) 


8.5 


0.1 


1.7 


Classified as disabled student (%) 


19.3 


13.2 


12.6 


Classified as economically or academically disadvantaged (%) 


51.6 


38.9 


37.3 


Characteristics of 9th-s>rade students 








Overage for grade 3 (%) 


62.2 


50.1 


45.2 


Attendance rate b (%) 


68.2 


72.6 


75.9 


Promoted to the 10th grade 


48.9 


61.7 


63.1 


Attendance rate greater than or equal to 90 percent 


27.4 


35.0 


42.8 


OPT test scores 0 








Reading comprehension 








Pass rates 


35.4 


50.1 


50.4 


Math total 








Pass rates 


13.7 


17.7 


23.3 


Total number of schools 


1 


4 


16 


(continued) 
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Table 4.1 (continued) 



SOURCE: MDRC calculations from individual student school records from Columbus Public Schools. 

NOTES: Sample consists of students for whom administrative records exist between the 1996-1997 and 2002-2003 
academic years. 

Cluster consists of a Project GRAD school matched with 4 comparison schools. Results in the Comparison 
Schools column reflect averages across these groups of non-Project GRAD schools. 

a A student is defined as overage for grade if he or she turns 15 before the start of the 9th grade. 
b Attendance rate is calculated by dividing the number of days present by the number of days enrolled. 
c OPT stands for Ohio Proficiency Test. 



school (Booker T. Washington High School) and at the comparison high schools chosen from 
the rest of the district. The table indicates that the vast majority of students served by both sets 
of schools — over 98 percent — were African-American. The Project GRAD and comparison 
high schools also served similar proportions of ninth-grade students who were overage for grade 
and who attended regularly. 2 

On the other hand, students at Washington High School scored slightly lower on both 
the reading and the math test than their counterparts at the comparison schools. While the aver- 
age normal curve equivalent score of the students at the comparison schools was 3 1 NCEs on 
the tenth-grade assessment in math in the years immediately preceding the intervention, the av- 
erage score of the students at the Project GRAD high school was 24 NCEs. 

These differences in baseline characteristics may reflect the fact that Project GRAD 
generally attempts to target the worst schools in the districts to which it expands. Although the 
patterns raise some concerns about the appropriateness of the comparison schools selected, the 
possible choices were restricted to the set of schools that existed in each district. In order to 
minimize the effect of these differences, the analysis focuses on the progress of each school 
relative to its own baseline outcomes, and it controls for individual student characteristics, in- 
cluding demographic traits and prior academic performance. 3 Moreover, the analysis does not 
directly compare the levels of student outcomes at the Project GRAD and the comparison 
schools; rather, it compares the degree of progress over time in each set of schools. 



2 The attendance rate in the Atlanta district is measured as the number of days present divided by 180 days. 

3 In Atlanta, the analysis controls for only race/ethnicity and being overage for grade. 
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The Project GRAD Evaluation 
Table 4.2 



Baseline Characteristics of Schools and Students at the Project GRAD and 
Comparison High Schools in Atlanta, 1997-1998 Through 1999-2000 



Characteristic 


Project GRAD 
School 


Comparison 

Schools 


All Atlanta 
High Schools 


Average school size 


9th grade 


549 


341 


431 


10th grade 


394 


256 


315 


1 1th grade 


307 


212 


257 


12 th grade 


224 


190 


221 


Total 


1,474 


999 


1,224 


Race/ethnicity (%) 


Black 


98.2 


99.6 


92.2 


White 


0.2 


0.2 


4.9 


Hispanic 


0.3 


0.2 


1.2 


Asian 


1.1 


0.0 


1.4 


Other 


0.0 


0.0 


0.1 


Gender (%) 


Male 


47.8 


48.9 


47.2 


Female 


52.2 


51.1 


52.8 


Classified as English speaker of other language (ESOL) (%) 


0.4 


0.0 


1.2 


Characteristics of 9th-srade students 


Overage for grade 3 (%) 


41.8 


43.1 


39.9 


Attendance rate b (%) 


67.6 


67.3 


71.5 


Attendance rate greater than or equal to 90 percent 


40.1 


34.3 


43.9 


ITBS test scores 3 
Reading total 


Average normal curve equivalent 11 (NCE) score 


23.6 


32.7 


31.1 


Math total 


Average normal curve equivalent 11 (NCE) score 


23.8 


30.3 


33.5 


Total number of schools 


1 


2 


11 



(continued) 



69 








Table 4.2 (continued) 



SOURCE: MDRC calculations from individual student school records from Atlanta Public Schools. 

NOTES: Sample consists of students for whom administrative records exist between the 1997-1998 and 2002- 
2003 academic years. 

Cluster consists of a Project GRAD school matched with 2 comparison schools. Results in the Comparison 
Schools column reflect averages across these groups of non-Project GRAD schools. 

a A student is defined as overage for grade if he or she turns 15 before the start of the 9th grade. 
b Attendance rate is calculated by dividing the number of days present by the number of days enrolled. 
C ITBS stands for Iowa Test of Basic Skills. 

d Like percentile scores, NCEs describe students' achievement relative to the national sample against which 
the test was normed. NCEs can range from 0 to 100, with a mean of 50 across the population against which the 
test was normed. Because it is valid to perform arithmetic operations only on cardinal measures, the analysis 
reported here is based on NCEs rather than on percentile measures of student achievement. 



Students’ Exposure to Project GRAD 

As explained in Chapter 1, rather than focusing on only one school at a time, Project 
GRAD implements reforms in a “feeder pattern’' of elementary and middle schools that “feed” 
students into a particular high school. The goal of this strategy is not only to target the high 
school in question but also to improve the academic preparation of students arriving at that high 
school. An examination of prior exposure to Project GRAD among ninth-graders in Atlanta and 
Columbus reinforces the fact that the program had not been in place long enough for the cumu- 
lative effects of the feeder-pattern strategy to come into play. Moreover, it also suggests that 
many students who entered the Project GRAD high schools in Columbus and Atlanta had not 
had any exposure to Project GRAD prior to entering high school. 

Figures 4.1 and 4.2 show students’ average number of years of exposure to Project 
GRAD prior to the ninth grade in Columbus and Atlanta, respectively. As Figure 4.1 illustrates, 
in the last year of the follow-up period, the average ninth-grader entering Linden-McKinley 
High School in Columbus had attended a Project GRAD elementary or middle school for ap- 
proximately a year and a half. In Atlanta, the average ninth-grader entering Washington High 
School had spent less than one year at a Project GRAD elementary or middle school. While the 
exposure of ninth-graders at the Project GRAD schools exceeded the exposure of ninth-graders 
at the comparison schools, these patterns suggest that — three to four years into the intervention 
— the majority of Project GRAD high school students in these two sites were not yet coming 
from elementary and middle schools in the Project GRAD feeder pattern. 
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The Project GRAD Evaluation 
Figure 4.1 

Columbus: Exposure to Project GRAD Prior to Ninth Grade 




School Years 



■ Project GRAD School □ Comparison Schools 



SOURCE: MDRC calculations from individual student school records from Columbus Public 
Schools. 

NOTES: Project GRAD began in Columbus in the 1999-2000 school year. 

Sample consists of students for whom administrative records exist between the 1996-1997 and 
2002-2003 academic years. 

Cluster consists of a Project GRAD school matched with 4 comparison schools. Results in the 
Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 



The Impacts of Project GRAD on Early High School Outcomes 

Project GRAD Columbus 

Project GRAD Columbus was first implemented at Linden-McKinley High School in 
the 1999-2000 academic year, and data are available through the spring of 2003. 
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The Project GRAD Evaluation 



Figure 4.2 

Atlanta: Exposure to Project GRAD Prior to Ninth Grade 




■ Project GRAD School □ Comparison Schools 



SOURCE: MDRC calculations from individual student school records from Atlanta Public Schools. 

NOTES: Project GRAD began in Atlanta in the 2000-2001 school year. 

Sample consists of students for whom administrative records exist between the 1997-1998 and 
2002-2003 academic years. 

Cluster consists of a Project GRAD school matched with 2 comparison schools. Results in the 
Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 



• Attendance rates among ninth-graders at the Project GRAD high school in 
Columbus improved after the program was implemented. 

• The improvements in attendance rates at the Project GRAD high school were mir- 
rored by similar changes at the comparison high schools throughout the district. 

Figure 4.3 shows the baseline average percentage of students who had “high atten- 
dance” — that is, attendance rates of 90 percent or greater — at both the Project GRAD and the 
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The Project GRAD Evaluation 
Figure 4.3 
Estimated Impacts: 

Percentage of Ninth-Graders in Columbus Who Had 
Greater Than or Equal to 90 Percent Attendance 
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SOURCE: MDRC calculations from individual student school records from Columbus Public Schools. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1996-1997 and 2002-2003 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline 
average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Cluster consists of a Project GRAD school matched with 4 comparison schools. Results in the Comparison 
Schools columns reflect averages across these groups of non-Project GRAD schools. 
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comparison high schools in Columbus. The figure also shows the deviations from the baseline 
average at the Project GRAD and comparison schools, as well as the estimated program effects 
(that is, the differences between the deviations from the baseline at the Project GRAD school 
and at its comparison schools). As in Chapter 3, the shaded bars represent deviations from the 
baseline average at the Project GRAD school, while the unshaded bars represent deviations 
from the baseline average at the group of comparison schools. 

In the years immediately preceding Project GRAD's implementation, the proportion of 
students who had greater than 90 percent attendance averaged 30 percent at the Project GRAD 
high school and 35 percent at the comparison schools. Over the first four years of Project 
GRAD, the proportion of students with high attendance increased by 26 percentage points, to 56 
percent at the Project GRAD high school. At the same time, the comparison schools experi- 
enced an average increase of 22 percentage points. While the 3.6 percentage point difference 
favors the Project GRAD schools, it is relatively small and not statistically significant. 

Project GRAD’s ultimate goal is to improve high school graduation rates and students’ 
success in college. Among the most important predictors of success and persistence in high 
school and of “on-time” graduation — which is a condition of receiving the Project GRAD col- 
lege scholarship — is promotion from ninth to tenth grade. 4 

• During the baseline period in Columbus, the ninth- to tenth-grade promotion 
rates at the Project GRAD high school were lower than the average rates at 
the comparison schools. 

• Promotion rates at the Project GRAD high school improved over the first 
few years of program implementation. Over the same period of time, promo- 
tion rates at the comparison schools showed a slight decline. 

• Although these differences are not statistically significant, they suggest a 
positive effect on ninth- to tenth-grade promotion rates. 

Figure 4.4 reports the average ninth- to tenth-grade promotion rates at baseline in Co- 
lumbus, as well as the deviations from the baseline at the Project GRAD and comparison high 
schools. As the figure indicates, in the three years prior to Project GRAD’s implementation, 
ninth- to tenth-grade promotion rates at Linden-McKinley High School averaged 53 percent, 



4 See Roderick (1993). 
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The Project GRAD Evaluation 
Figure 4.4 
Estimated Impacts: 

Percentage of Ninth-Graders in Columbus Who Were Promoted to Tenth Grade 
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Year 2 Year 3 

Follow-Up Years 

Baseline Average: Project GRAD School = 53.1 Comparison Schools = 61.9 



I Project GRAD School 



□ Comparison Schools 



SOURCE: MDRC calculations from individual student school records from Columbus Public Schools. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1996-1997 and 2002-2003 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline 
average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Cluster consists of a Project GRAD high school matched with 4 comparison schools. Results in the 
Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 
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compared with 62 percent at the comparison schools. Although the promotion rates fluctuated 
over the course of the follow-up period, during the first three years of the program, there was a 
net growth of 9 percentage points — to 62 percent — in promotion rates at the Project GRAD 
Columbus high school. At the same time, promotion rates at the comparison schools fell 4 per- 
centage points, to 58 percent. 

This difference suggests that Project GRAD had a positive impact of 13 percentage 
points on ninth- to tenth-grade promotion rates at Linden-McKinley High School. These differ- 
ences are not statistically significant, however, so one cannot rule out the possibility that they 
are driven by chance. 5 

Project GRAD Atlanta 

Project GRAD was implemented at Booker T. Washington High School in Atlanta in 
the 2000-2001 academic year. Inasmuch as data are available through 2002-2003, the analysis 
can follow the program’s effects through three years of implementation. 

• The Project GRAD high school in Atlanta has seen improvements in both at- 
tendance rates and ninth- to tenth-grade promotion rates. 

• In general, though most of these differences are not statistically significant, 
these improvements appear larger than the changes at the comparison schools. 

Figure 4.5 shows baseline attendance rates and the degree of progress over time at the 
Project GRAD and comparison high schools in Atlanta. The patterns suggest that Project 
GRAD had a positive impact on attendance, at least through the second year of follow-up. In 
the baseline period, an average of 41 percent of Project GRAD ninth-graders had 90 percent 
attendance or better, compared with about 35 percent at the comparison schools. In the first two 
years of the program, the improvements at the Project GRAD school clearly outpaced progress 
at the comparison schools. From the baseline period to two years after program implementation, 
the percentage of ninth-grade students at Washington High School who had high attendance 
increased from below 41 percent to above 53 percent — a change of nearly 13 percentage 
points. During the same period, the percentage of ninth-grade students who had high attendance 
rates at the comparison schools increased by only 1 percentage point. This results in a statisti- 
cally significant impact of approximately 12 percentage points. In the third year of follow-up, 
the percentage of ninth-graders at the Project GRAD school who had high attendance continued 



5 The fact that a difference of this magnitude is not statistically significant underscores the limitations of 
the statistical inferences that can be made from analyses involving only one Project GRAD school. 
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The Project GRAD Evaluation 
Figure 4.5 
Estimated Impacts: 

Percentage of Ninth-Graders in Atlanta Who Had 
Greater Than or Equal to 90 Percent Attendance 
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Baseline Average: Project GRAD School = 40.7 Comparison Schools = 35.4 



■ Project GRAD School □ Comparison Schools 



SOURCE: MDRC calculations from individual student school records from Atlanta Public Schools. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1997-1998 and 2002-2003 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline 
average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from baseline between Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Cluster consists of a Project GRAD high school matched with 2 comparison schools. Results in the 
Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 
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to increase — to 77 percent. The percentage of such students at the comparison schools also 
grew, to 64 percent. So even though the Project GRAD and comparison schools made similar 
progress in terms of attendance rates, it appears that Project GRAD created these results sooner 
than they otherwise would have occurred. 

Figure 4.6 shows baseline averages and program effects on ninth- to tenth-grade promo- 
tion rates in Atlanta. During the baseline period, an average of 59 percent of ninth-graders at the 
Project GRAD high school in Atlanta were promoted to the tenth grade by the next school year. 
The proportion of Project GRAD ninth-graders who were being promoted to the tenth grade 
appears to have grown in the first two years of follow-up. 6 In both years, the gains at the Project 
GRAD school appear to have exceeded the gains at the comparison schools. Yet these differ- 
ences are not statistically significant, and one cannot rule out the possibility that they were 
driven by chance. 



Conclusions 

Though not conclusive, these results suggest that Project GRAD may have had a posi- 
tive influence on the precursors to graduation in two of the initiative’s early expansion sites: 
Columbus and Atlanta. The estimated program effects in these sites are often not statistically 
significant. Nevertheless, the comparison of the progress made at Project GRAD high schools 
and at similar comparison schools from the same districts consistently favors the Project GRAD 
schools. Given the small sample sizes in these analyses, the absence of statistical significance 
could be driven by a lack of statistical power rather than by a lack of systematic effects. These 
results are both modest and early, measuring only the first few years of program implementa- 
tion. As such, they should be interpreted with caution. Key questions include whether these dif- 
ferences will increase or wane in the next several years of Project GRAD' s implementation — 
and whether they will translate into meaningful effects on high school students’ graduation rates 
and preparation for college. 



6 Although there are three years of data for other outcomes in Atlanta, the analysis can calculate promotion 
rates for only two years after program implementation because data from the following academic year are 
needed in order to determine promotion status. 
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The Project GRAD Evaluation 
Figure 4.6 
Estimated Impacts: 

Percentage of Ninth-Graders in Atlanta Who Were Promoted to Tenth Grade 
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■ Project GRAD School □ Comparison Schools 



SOURCE: MDRC calculations from individual student school records from Atlanta Public Schools. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1997-1998 and 2002-2003 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline average 
and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from baseline between Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Cluster consists of a Project GRAD school matched with 2 comparison schools. Results in the Comparison 
Schools columns reflect averages across these groups of non-Project GRAD schools. 
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Chapter 5 

Conclusions and Implications for Project GRAD 



High school reform — the focus of increased attention among policymakers and practi- 
tioners — is in many ways more challenging than improvement efforts that are targeted at lower 
grades. For instance, elementary schools are usually smaller and simpler institutions than high 
schools; they cover a narrower range of academic subjects and student skill levels, and more is 
known about effective ways to guide elementary instruction. The contrast between recent im- 
provements in student success at many low-performing elementary schools and the slow and 
inconsistent progress at the high school level only highlights the challenge that Project GRAD 
faces in working with troubled high schools. 

Recognizing the difficulty of improving student achievement at the high school level, 
Project Graduation Really Achieves Dreams (GRAD) — like many reform efforts — has paid 
close attention to trends in student outcomes, documenting these measures as operational indica- 
tors at the schools that have adopted the program. By participating in this evaluation, however, 
Project GRAD has also tackled a strategically important and more demanding question: How do 
the trends in student outcomes at the Project GRAD schools compare with the trends at similar 
schools using other strategies? Project GRAD sought to understand its contribution to improving 
student outcomes above and beyond other strategies — as well as to identify aspects of its ap- 
proach that need refinement. This report and the companion study on elementary schools use the 
shorthand of impacts, or effects, to describe the added value that Project GRAD brings. 1 

The very nature and complexity of the Project GRAD feeder system intervention, 
which posits that students would need to be exposed to the program over many years, combined 
with the limited amount of follow-up in the expansion sites and the reform-rich context of urban 
school districts, created a challenging set of conditions for a meaningful evaluation. While the 
Houston feeder patterns provide a reasonable test of the intervention, the results for Atlanta and 
Columbus should be treated as more provisional. In addition, because the high school evalua- 
tion began concurrently with the implementation of Project GRAD in the expansion site feeder 
elementary and middle schools, the study was not in a position to capture cumulative effects of 
students’ exposure to the intervention in the earlier grades in those districts. Only in the latter 
years of follow-up in Houston did this begin to be possible. Finally, findings from the single 
high schools in Atlanta and Columbus — as well as from Jefferson Davis High School in Hous- 
ton — should be interpreted with some caution. 



'For information about Project GRAD at the elementary level, see Snipes, Holton, and Doolittle (2006). 
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What Is Unique About Project GRAD? 

As described in this report, Project GRAD differs from other high school reform efforts 
in several other important ways. First, rather than focusing solely on a single high school, Pro- 
ject GRAD also intervenes in the feeder pattern of elementary and middle schools that are asso- 
ciated with that high school. In this way, it straddles the gap between district-level attempts to 
reform all schools and whole-school reforms that work with individual schools. Second, Project 
GRAD has an unusual theory of change that guides its design. Many high school reforms mod- 
ify classroom instruction through distinctive curricular offerings intended to address the skill 
deficits of incoming students, offer teachers intensive professional development on content and 
pedagogy, and/or restructure schools into smaller learning communities. In contrast, Project 
GRAD's strategy in high schools relies on the demand push for change from better-prepared 
entering students coming from Project GRAD feeder schools and the opportunity pull toward 
success provided by the promise of financial support for college through the Project GRAD 
scholarship offer. To bolster the effects of these forces, Project GRAD adds classroom man- 
agement and social service components at the high school level to improve the learning envi- 
ronment and to address the motivation of students, focusing their attention on college-going and 
supporting their progress through high school and college. Finally, Project GRAD adopts a de- 
cidedly long view of high school reform, arguing that making a sustained investment in improv- 
ing the core academic skills of students before they enter ninth grade will have a bigger payoff 
on high school performance than addressing skill deficits after students reach high school. 

Project GRAD’s model assumes that better-prepared students would come from the 
Project GRAD feeder schools, would benefit from special academic counseling and summer 
academic enrichment in high school, and would qualify for a scholarship to attend college, 
which is the “cornerstone” of the Project GRAD reform. The combination of Project GRAD 
reforms at the elementary school level — which directly affect instruction in classrooms — was 
intended to increase the chances that students could take advantage of the opportunity provided 
by the college scholarship. At the high school level, there has not been a similar effort to change 
instruction through structural changes in the organization of the school, new curricula to address 
skill deficits, or intense professional development. 



What Do These Research Findings Suggest? 

The findings in this report — combined with the companion elementary school study, 
other research, and operational and policy knowledge — provide insights into the strengths that 
Project GRAD brings to school improvement efforts as well as into the challenges that it still 
faces. MDRC’s elementary school evaluation found that students at Project GRAD schools 
generally showed as much improvement on high-stakes state achievement tests as students at 
similar local schools — but without suffering the decline seen in comparison schools on na- 
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tional tests, which measure achievement more broadly. This suggests that Project GRAD can 
modestly improve the academic achievement of students leaving elementary school — an es- 
sential element of its strategy to develop better-prepared students entering high school. 

However, the findings from the high school study tell a more complicated story. The 
evaluation found only inconsistent effects on academic success among ninth- and tenth-graders 
and no effects on the percentage of ninth-graders who graduated from the three longest- 
operating Project GRAD high schools studied. Yet among those who did graduate from Project 
GRAD's flagship, Davis High School in Houston, students were more likely to have completed 
a core academic curriculum within four years, meaning that they were better prepared to enter 
college. In the two other Houston high schools studied, these effects have not yet appeared, and 
it is too early to detect such trends in Atlanta and Columbus, the expansion sites in the study. 

Why does this pattern of findings appear? The evidence points to at least two hypothe- 
ses. First, it may be more difficult than expected to quickly improve the academic performance 
of incoming high school students by intervening in feeder schools — a core tenet of Project 
GRAD's strategy. The implementation research highlights the fact that the Project GRAD strat- 
egy takes time to unfold; in each site, the various components at all schools were rolled out 
gradually. In addition, it turns out that the feeder patterns for the high schools examined in this 
study were “leaky” — as a result of high rates of mobility and school-choice options, many stu- 
dents in the Project GRAD high schools had not benefited from exposure to the model in ele- 
mentary or middle school . 2 

Second, other research has zeroed in on the crucial transitions that students must make 
as they enter high school, particularly on the central role that completing ninth grade on time 
plays in a student’s eventual completion of high school . 3 Being “on track” at the end of the first 
year of high school is a stronger predictor of eventual on-time graduation than a student’ s enter- 
ing achievement level is . 4 Thus, because early indicators of high school success are important, 
they are featured as key outcomes in this report. One explanation for the lack of impacts by Pro- 
ject GRAD on these ninth- and tenth-grade measures (and perhaps on graduation rates) could be 
the initiative’s lack of a direct intervention in ninth-grade instruction during the time period 
covered by the study. On the other hand, while Project GRAD did not produce improvements in 
graduation rates, its services were able to affect the course-taking of students who were already 
headed toward graduation. The scholarship requirements, counseling on the best academic 



“Research in other settings — for example, Chicago — illustrates that it is possible to improve academic 
outcomes for high school students by improving academic performance in the early grades through a district- 
wide effort. See Roderick (2006). 

Roderick (2006); Quint (2006). 

Roderick (2006). 
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preparation for college, and social service supports seem likely reasons for the impacts on the 
academic preparation of graduates at the flagship high school in Houston. 

It may well be that the larger problems facing most low-performing high schools over- 
whelmed the combination of the “demand push” and the “opportunity pull” of Project GRAD's 
strategy. Students in many high schools encounter large impersonal organizations, which are 
focused more on academic content and less on building individualized relationships among stu- 
dents and teachers. Course offerings are often driven by graduation requirements, meaning that 
the development of new and better curricula can be a slow process. So, absent specific efforts 
and new resources, the demands of the better-prepared entering students may not be enough to 
spark necessary change in high school instruction. 



What Are the Implications of These Findings for Project GRAD? 

Given Project GRAD’s long-view strategy of developing better-prepared students in 
feeder schools, this evaluation of its high school intervention has come relatively early in its 
development, particularly in the expansion sites outside its home district of Houston. At the 
same time, the marketplace in which Project GRAD operates is highly competitive. Local deci- 
sion-makers, who face intense pressure to improve high schools quickly, are weighing Project 
GRAD against other reforms that intervene more directly in the daily educational experience of 
high schools — many of which have not taken on a comparative study like this to understand 
their own added value. 

One of the key implementation findings of MDRC’s study is that Project GRAD is a 
dynamic organization that has responded to operational lessons, changing policy emphases, and 
research evidence to modify its strategy over time. In fact, it has already begun to refine its high 
school approach to address some of the challenges suggested by this evaluation, including the 
effects of “leaky” feeder patterns, the relatively slow rollout of the components of the Project 
GRAD intervention, and the difficulties of transforming high schools without directly interven- 
ing in classrooms. Several refinements — some of which Project GRAD has begun to adopt — 
seem promising. 

First, Project GRAD continues to enhance the learning environments in its high 
schools. For years, Project GRAD has included components to improve classroom and school 
climate and functioning (Consistency Management & Cooperative Discipline^), address per- 
sonal and family issues that hamper learning (Communities In Schools and, more recently, 
Campus Family Support), and create better counseling and support for college-going. It has 
long operated in tandem with other reforms in its high schools to offer complementary services. 
Recently, Project GRAD has collaborated with district efforts to create smaller structures within 
schools — small learning communities — that can provide more personalized and more orderly 
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learning environments. (Recent research suggests that small learning communities can provide a 
setting in which it easier to address other educational concerns.) 5 

Second, directly addressing the skill deficits of entering high school students through 
curricular innovations could improve the percentage of ninth-graders making a successful tran- 
sition to tenth grade. Several complementary high school reforms include special ninth-grade 
coursework that focuses on developing key gateway skills (like algebra) to later subjects and 
study strategies that carry across all academic topics. There is growing evidence that changing 
ninth-grade instruction in these ways can produce improvements in high school outcomes. 6 In a 
sense, Project GRAD may face the choice of working in high schools where new strategies are 
already in place to improve instruction and surrounding these efforts with the support it offers 
or partnering with other organizations to add instructional reforms to the package Project 
GRAD brings to the schools it serves. 

Finally, recent experience suggests that organizations outside school districts can be 
important in stimulating and supporting high school improvements. Project GRAD has already 
traveled a long way in building this kind of capacity for participating districts. Over the last dec- 
ade, Project GRAD has evolved from an effort in one feeder pattern in Houston to a broader 
initiative throughout the district, to a coalition of affiliated efforts across several districts, to a 
network of 18 feeder patterns in 12 districts supported by local Project GRAD organizations 
and a substantial national organization, Project GRAD USA. As the number of high school re- 
form models multiplies nationally, the challenge for the field is to develop ways for different 
reforms to work together. It will be increasingly rare for any single reform organization to be 
given free rein to work in a troubled high school or district over an extended period. More and 
more, schools and districts are seeking to integrate various reform approaches with complemen- 
tary strengths that will serve the needs of their students. 

In many ways, Project GRAD's multifaceted strategy was ahead of its time, prefiguring 
a variety of current reform approaches. The focus on the full span of grades, the connection to 
postsecondary education, and the need to work above the level of individual schools are now 
appreciated as important aspects of many district-level reforms. Project GRAD now has the op- 
portunity to build on its strengths, incorporate additional components into its strategy, and de- 
velop strategic partnerships with other complementary school improvement efforts — to create 
the next generation of its reform model. 



5 Quint (2006). 

6 Kemple, Herlihy, and Smith (2005). 
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Appendix A 

The Impacts on High School Graduation 
Among All Ninth-Grade Students in Houston 




The estimated effects of Project Graduation Really Achieves Dreams (GRAD) on high 
school graduation rates that are presented in the body of this report focus on outcomes for first- 
time ninth-graders. However, as is the case in many high schools throughout the country, in any 
given year, a substantial portion of ninth-graders who were enrolled at the Project GRAD high 
schools and their comparison schools in Houston were students who were repeating the ninth 
grade. It is therefore possible that the effects of Project GRAD on graduation-related outcomes for 
first-time ninth-graders are different from the effects of the program on ninth-graders as a whole. 

In order to explore this possibility, this appendix reports the estimated effects of Project 
GRAD on high school graduation for the full sample of ninth-grade students at Davis, Yates, 
and Wheatley high schools. In short, although the graduation rates are lower for the full sample 
of ninth-graders than for the first-time ninth-graders, the pattern of impacts is similar. 

Appendix Figure A.l presents the distribution of graduation rates among all ninth- 
graders at the Project GRAD high schools in Houston during the years immediately preceding 
Project GRAD’s implementation at each school. As the figure shows, the majority of entering 
ninth-graders did not graduate within four years. In fact, over 50 percent of them were no longer 
enrolled in the district three years later. About 54 percent of the ninth-graders in the sample left 
the district — that is, either dropped out or moved to another district — before their scheduled 
end of the twelfth grade; 14 percent remained enrolled in the district but had not reached the 
twelfth grade three years later; and 10 percent made it to the twelfth grade but did not graduate 
on time. Even among those ninth-graders who reached the twelfth grade on time, about a quar- 
ter of students did not graduate on time. In the end, an average of 22 percent of all ninth-graders 
in Houston’s Project GRAD high schools graduated three years later. 

Appendix Figure A.2 presents the estimated impacts of Project GRAD on graduation 
rates among all ninth-grade students in Houston. It addresses the question: Did a student ever 
graduate from the Houston Independent School District (HISD)? The upper panel of the figure 
presents the estimated impacts of Project GRAD at Davis High School, while the lower panel 
presents the average impacts of Project GRAD across Davis, Yates, and Wheatley high schools 
combined. As in the body of the report, the baseline averages for the Project GRAD and the 
comparison schools appear below each graph. 

The vertical bars in the graph represent the difference between these baseline averages 
and the average outcomes in each of two parts of the follow-up period: the first three years after 
Project GRAD’s implementation (Years 1-3) and the next three years (Years 4-6). The shaded 
bars represent the deviation from the baseline average at the Project GRAD school(s), and the 
unshaded bars represent the deviation from the baseline average at the comparison schools. The 
figure also reports the difference between the deviation from the baseline at the Project GRAD 
school and the deviation from the baseline at the comparison schools. This is the estimated pro- 
gram “impact,” that is, the estimated effect of Project GRAD over and above the changes that 
would have been observed without the program. 
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The Project GRAD Evaluation 
Appendix Figure A.l 

Houston: Distribution of All Ninth-Graders Who Graduated On Time 
Three Years Later, Average Prior to Project GRAD 




SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: The baseline period for Davis High School is 1991-1992 to 1993-1994. The baseline period for Yates 
High School is 1993-1994 to 1995-1996. The baseline period for Wheatley High School is 1994-1995 to 1996- 
1997. 

Baseline averages are unadjusted. 
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The Project GRAD Evaluation 
Appendix Figure A.2 

Houston: Percentage of All Ninth-Graders Who Ever Graduated, 
Six-Year Follow-Up Results 
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Appendix Figure A.2 (continued) 

SOURCE: MDRC calculations from individual student school records from the Houston Independent School 
District. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 
academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline 
average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD 
schools and the "deviation from the baseline" for the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and 
comparison schools. Standard errors and statistical significance levels of deviations from the baseline are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. 
Results in the Comparison Schools columns reflect averages across these groups of non-Project GRAD schools. 

This figure focuses on analysis of successive cohorts of first-time ninth-graders at Project GRAD and 
comparison schools and whether students graduated on time three years later. 



The patterns of findings on these outcomes are similar to the patterns presented in the 
report for a sample of first-time ninth-grade students. The estimated program effects on gradua- 
tion rates that are presented in the report focus on whether or not first-time ninth-grade students 
graduated within four years of starting high school. However, many students who graduate do 
so in more than four years. Therefore, it is possible that focusing only on “on-time” graduation 
could generate inaccurate estimates of graduation effects. 1 With that in mind, Appendix Figure 
A.2 presents the estimated effects on graduation rates regardless of whether or not students 
graduated on time. In other words, for each class (cohort) of ninth-graders, the analysis calcu- 
lates the percentage of students who graduated in any year of the follow-up period (which ended 
in 2004 but includes graduates only through 2003). As is the case for four-year graduation rates, 
the observable trend is an increase in the percentage of ninth-grade students graduating at both 
the Project GRAD and the comparison high schools. In general, the differences between the 
progress at Davis High School and its comparison counterparts are relatively small (less than 1 
percentage point by the end of the follow-up period) and are not statistically significant. Inter- 
estingly, the three Project GRAD high schools as a whole experienced a smaller increase in 
graduation rates than their comparison schools, resulting in a modest but statistically significant 
negative effect on graduation rates. 

'The effects could be overestimated or underestimated, depending on the dynamics at work. 
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Appendix B 

High School Achievement in Houston: 
Was There Shifting of the Pool of Test-Takers? 




The body of this report presents the estimated effects of Project Graduation Really 
Achieves Dreams (GRAD) on ninth-grade students who subsequently took and passed the 
tenth-grade Texas Assessment of Academic Skills (TAAS) at three Project GRAD Houston 
high schools (Davis, Yates, and Wheatley), which is contingent on a ninth-grade student’s get- 
ting to the tenth grade to take the test by the following year. Despite rising pass rates on the 
math and reading portions of the TAAS at both the Project GRAD high schools and the com- 
parison high schools, generally, there were inconsistent, modest, and not statistically significant 
effects on student achievement. 

Simply looking at pass rates for ninth-graders on the tenth-grade test might be subject to 
a problem with gaming of the test-taking population. Because of this concern, the body of this 
report presents a measure that includes both taking the test and passing the test. Given the pres- 
ence of high-stakes accountability systems in Texas and the great importance of the tenth-grade 
test, theoretically, it is possible that schools were holding back lower-performing ninth-graders 
in the ninth grade and skipping them to the eleventh grade, in order to artificially increase pass 
rates among the tenth-graders who took the test. If this practice was common — and if it was 
more prevalent among the comparison schools than the Project GRAD schools — it could have 
generated biased estimates of program effects. In particular, by reducing the number of lower- 
performing students who took the test, the comparison schools could have overstated the 
achievement levels — or the growth in achievement — of their students. If the Project GRAD 
schools were not doing the same thing, the disparate growth in achievement at the comparison 
schools might suggest negative program effects where none exist, by systematically shifting the 
test-taking population of tenth-graders. 

By looking only at students who took and passed the tenth-grade test (as is done in 
Chapter 3), this measure already includes a component that reflects differential promotion or 
test-taking patterns. This appendix further explores this issue, first by reviewing the implications 
of estimated effects on promotion rates and then by presenting the results of several additional 
sensitivity analyses. 

With respect to promotion rates, to the extent that the comparison schools systemati- 
cally manipulated promotions in ways that were different from the Project GRAD schools, this 
should have generated impacts on achievement. In particular, if fewer lower-performing ninth- 
grade students were promoted to tenth grade and then skipped to eleventh grade the following 
year, the ninth- to tenth-grade promotion rates should be reduced. To the extent that this practice 
occurred at the comparison schools but not at the Project GRAD schools, the program should 
have had positive effects on ninth- to tenth-grade promotion rates. Looking across the estimates 
at Davis, Yates, and Wheatley High Schools suggests relatively modest, statistically insignifi- 
cant effects on promotion — for example, a positive effect of less than 5 percentage points in 
the second three years of implementation. During that same time, the estimated effects on pass 
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rates on the math portion of the TAAS suggest an effect of negative 3 percentage points. 
Though not impossible, it seems unlikely that the difference in achievement was driven by the 
difference in promotion rates. 

Nevertheless, MDRC employed several other sensitivity tests to explore the extent to 
which there was any gaming of the test-taking population that resulted in different test-taking 
patterns and whether these differences could have resulted in biased estimates of program ef- 
fects on achievement. First, the analysis explores the percentage of ninth-grade students in a 
given year who did not take the tenth-grade test the following year. The findings suggest that 
the vast majority of ninth-graders did not take the test the following year and that there were 
only small differences in the changes over time in test-taking patterns at Davis, Yates, and 
Wheatley High Schools versus their comparison counterparts. 

Appendix Table B.l reports the estimated effects of the percentage of ninth-grade stu- 
dents who did not take the tenth-grade math portion of the TAAS (anywhere in the Houston 
Independent School District) the following academic year. Perhaps the most striking finding in 
this table is that, in any given year, the majority of ninth-graders at both Project GRAD and 
comparison schools did not take the tenth-grade math test the following year. For example, dur- 
ing the baseline period, roughly 73 percent of the ninth-graders at Davis, Yates, and Wheatley 
High Schools combined did not take the tenth-grade math portion of the TAAS the following 
academic year. The same could be said for approximately 72 percent of the ninth-graders at the 
comparison schools. Appendix Table B.l also suggests that the percentage of test-takers grew 
by roughly 2 percentage points at the Project GRAD high schools and by 4 percentage points at 
the comparison schools. In other words, though the difference is not statistically significant, the 
increase in test-takers was greater at the comparison schools than at the Project GRAD schools 
— the opposite of what one would expect if gaming were driving the test results. Appendix Ta- 
ble B.2 presents parallel estimates for the percentage of ninth-graders not taking the tenth-grade 
reading portion of the TAAS the following year, and it shows similar results. 

In short, this analysis does not suggest that the comparison schools were testing a 
smaller proportion of their ninth-grade students or that test-taking patterns were changing in a 
way that would generate favorable achievement results at the comparison schools. 
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The Project GRAD Evaluation 
Appendix Table B.l 



Interrupted Time Series and Impact Estimates for Ninth-Graders Not Taking the Tenth-Grade Math Test 
One Year Later, Follow-Up Results, by Houston Feeder Pattern 





Project GRAD Schools 




Comparison Schools 




Impact 


Impact Effect Size 11 


Feeder Pattern 


Baseline 


Years 1-4 Years 5-7 


Baseline 


Years 1-4 Years 5-7 


Years 1-4 Years 5-7 


Years 1-4 Years 5-7 


Davis 

lOth-grade math test 


















Pass rate 


61.7 


70.0 


62.0 


66.0 


72.7 


69.3 






Deviation from baseline average 




8.3 


0.3 




6.7 


3.3 


1.6 -3.0 


0.04 -0.07 


All clusters 

lOth-grade math test 


















Pass rate 


72.7 


70.8 




72.4 


68.3 








Deviation from baseline average 




-1.9 






-4.1 




2.2 


0.05 



SOURCE: MDRC calculations from individual student school records front the Houston Independent School District. 



NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation front the baseline" for Project GRAD schools and the "deviation from the baseline" for 
the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and comparison schools. Standard errors and statistical 
significance levels of deviations from the baseline are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. Results in the Comparison Schools columns reflect 
averages across these groups of non-Project GRAD schools. 

In 2002 the test changed front the TAAS to the TAKS. 

Ninth-graders did not take the lOth-grade math test the next year. 

"The "impact effect size" was calculated by dividing the "impact" by the standard deviation of outcomes for all 9th-grade students in the district's non- 
selective comprehensive high schools during the 1993-1994 through 1995-1996 school year. 




The Project GRAD Evaluation 
Appendix Table B.2 



Interrupted Time Series and Impact Estimates for Ninth-Graders Not Taking the Tenth-Grade Reading Test 
One Year Later, Follow-Up Results, by Houston Feeder Pattern 





Project GRAD Schools 




Comparison Schools 




Impact 


Impact Effect Size" 


Feeder Pattern 


Baseline 


Years 1-4 Years 5-7 


Baseline 


Years 1-4 Years 5-7 


Years 1-4 Years 5-7 


Years 1-4 Years 5-7 


Davis 

lOth-grade reading test 
Pass rate 


62.1 


70.2 


62.4 


66.5 


73.3 


69.6 






Deviation from baseline average 




8.1 


0.2 




6.8 


3.1 


1.2 -2.9 


0.03 -0.07 


All clusters 
lOth-grade reading test 
Pass rate 


73.0 


71.2 




72.8 


69.2 








Deviation from baseline average 




-1.8 






-3.6 




1.8 


0.04 



SOURCE: MDRC calculations from individual student school records from the Houston Independent School District. 

NOTES: Estimates are regression-adjusted for students' background characteristics. 

Sample consists of students for whom administrative records exist between the 1990-1991 and 2003-2004 academic years. 

The "deviation from the baseline" for each year was calculated as the difference between the baseline average and the average for the specified year. 

The "impact" was calculated as the difference between the "deviation from the baseline" for Project GRAD schools and the "deviation from the baseline" for 
the comparison schools. 

A two-tailed t-test was applied to differences in deviations from the baseline for Project GRAD and comparison schools. Standard errors and statistical 
significance levels of deviations from the baseline are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** 

= 5 percent; * = 10 percent. 

Clusters consist of a Project GRAD school matched with a group of between 2 and 4 comparison schools. Results in the Comparison Schools columns reflect 
averages across these groups of non-Project GRAD schools. 

In 2002 the test changed from the TAAS to the TAKS. 

Ninth-graders did not take the lOth-grade reading test the next year. 

"The "impact effect size" was calculated by dividing the "impact" by the standard deviation of outcomes for all 9th-grade students in the district's non- 
selective comprehensive high schools during the 1993-1994 through 1995-1996 school year. 




Appendix C 

Selecting Comparison Schools 




Comparison schools are included in the evaluation of Project Graduation Really Achieves 
Dreams (GRAD) in order to build on the information already provided in the baseline patterns at 
Project GRAD schools. Combined with the information regarding student achievement at base- 
line, information from the comparison schools provides an estimate of the outcomes that would 
have been observed at the program schools in the absence of the Project GRAD intervention. Spe- 
cifically, the comparison schools refine the estimates of how student outcomes at the program 
high schools would have changed in the absence of Project GRAD. For example, suppose student 
outcomes improved dramatically at the comparison schools. If these schools were from the same 
district, were similar to the Project GRAD high schools in terms of students' demographic charac- 
teristics, and demonstrated a similar history of student outcomes, it may be reasonable to conclude 
that — even in the absence of Project GRAD — the program schools might have experienced a 
similar amount of progress. On the other hand, if the outcomes at a set of comparison schools de- 
clined after the baseline period, the most reasonable conclusion may be that — absent the Project 
GRAD intervention — outcomes at the program schools would have worsened. 

If the comparison schools are to serve this purpose, they must be as similar as possible 
to the program schools in the analysis. Therefore, the goal in choosing comparison schools is to 
find a set of high schools that, absent the intervention, would be expected to perform in a man- 
ner similar to that of the Project GRAD schools. 

In order to ensure “face validity” as well as to guard against the possibility that schools 
serving different populations of students could evolve differently in response to the same local 
events, the analysis limits comparison schools to those that served demographically similar stu- 
dent bodies. In order to further limit the set of comparison schools to those that had a similar 
track record in terms of student outcomes, the analysis also limits the comparison schools to 
those that had similar histories of prior achievement. In particular, for each Project GRAD high 
school in Houston, the selected set of comparison schools from the same school district met the 
following criteria: 

• The percentage of students in the modal racial/ethnic group at the Project 
GRAD high school was within 20 percentage points of the percentage of stu- 
dents in that racial/ethnic group at the comparison school. 

• Average TAAS scores in reading and math during the baseline period were 
within 0.25 standard deviation of average TAAS scores at the Project GRAD 
high school. 

Table 3.2 (in Chapter 3 of the report) presents the average baseline characteristics of stu- 
dents at the three Project GRAD Houston high schools included in this analysis (Davis, Yates, and 
Wheatley) and the baseline characteristics of students at their comparison counterparts. The table 
shows that, on average, the comparison high schools chosen for the analysis served similar — 
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though not identical — student populations and exhibited similar levels of student outcomes dur- 
ing the years immediately preceding Project GRAD’s implementation at each high school. 

For example, the table shows that the Project GRAD schools served slightly more black 
students than their comparison counterparts. On average, 56 percent of the students served by 
the Project GRAD schools during the baseline period were black, compared with 44 percent at 
the comparison schools. The schools served similar proportions of Hispanic students, 42 percent 
at both the Project GRAD and the comparison schools. So, in total, 98 percent of the students 
served by the Project GRAD schools were black or Hispanic, compared with 86 percent at the 
comparison schools. It’ s important to note that these differences are modest and that measures 
of students’ race/ethnicity are included in the equations predicting the program’s impacts. 

It is worth noting that — prior to the implementation of Project GRAD — the students 
at the Project GRAD and comparison schools exhibited similar outcomes. For example, 61 per- 
cent of students at the Project GRAD schools had attendance rates of 90 percent or greater, 
compared with 62 percent at the comparison schools. At both the Project GRAD and the com- 
parison high schools, approximately 50 percent of ninth-graders were promoted to the tenth 
grade. Tenth-grade pass rates on the reading portion of the TAAS averaged 61 percent at the 
Project GRAD schools, compared with 63 percent at the comparison schools. The tenth-grade 
pass rates on the math portion of the TAAS averaged 44 percent at the Project GRAD schools, 
compared with 42 percent at the comparison schools. 

In general, this suggests that though there were slight differences in students' demo- 
graphic characteristics, the Project GRAD and comparison schools served similar student popula- 
tions and exhibited similar student outcomes in the years immediately preceding Project GRAD. 

Because Chapter 3 of this report presents impact estimates separately for Davis High 
School, Table 3.1 shows the baseline characteristics of students there and at its comparison 
schools. As the table suggests, the vast majority of students at Davis and its comparison schools 
were Hispanic, but this percentage was somewhat larger at Davis — 84 percent versus 73 per- 
cent. As is the case for the aggregate numbers, Davis High School students and their compari- 
son school counterparts exhibited similar outcome patterns during the baseline period. 

In general, the program schools and the comparison schools in this analysis exhibited 
slight differences in demographic characteristics (which were controlled for in the analysis of 
program effects), but they were similar in terms of student outcomes in the years preceding the 
implementation of Project GRAD. 

As is the case for the Houston analysis, the comparison schools in Atlanta and Colum- 
bus were chosen on the basis of similarity to the Project GRAD students’ demographic charac- 
teristics and achievement outcomes. In particular, in both districts, comparison schools were 
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limited to those that were within 20 percentage points of the modal racial/ethnic groups at the 
Project GRAD high school and were within 20 percentage points of that school’ s average base- 
line pass rates in reading and math. Table 4.1 compares the characteristics of students at Linden- 
McKinley High School in Columbus with the comparison schools chosen for the analysis. Ta- 
ble 4.2 shows the characteristics and outcomes among students at Booker T. Washington High 
School in Atlanta versus the comparison schools chosen from that district. As in Houston, these 
Project GRAD high schools and their comparison schools were similar — though not identical 
— in terms of students’ baseline characteristics and baseline achievement outcomes. 
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About MDRC 



MDRC is a nonprofit, nonpartisan social and education policy research organization 
dedicated to learning what works to improve the well-being of low-income people. 
Through its research and the active communication of its findings, MDRC seeks to en- 
hance the effectiveness of social and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best 
known for mounting rigorous, large-scale, real-world tests of new and existing policies 
and programs. Its projects are a mix of demonstrations (field tests of promising new pro- 
gram approaches) and evaluations of ongoing government and community initiatives. 
MDRC’s staff bring an unusual combination of research and organizational experience to 
their work, providing expertise on the latest in qualitative and quantitative methods and 
on program design, development, implementation, and management. MDRC seeks to 
learn not just whether a program is effective but also how and why the program’s effects 
occur. In addition, it tries to place each project’ s findings in the broader context of related 
research — in order to build knowledge about what works across the social and education 
policy fields. MDRC’s findings, lessons, and best practices are proactively shared with a 
broad audience in the policy and practitioner community as well as with the general pub- 
lic and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of pol- 
icy areas and target populations. Once known primarily for evaluations of state welfare- 
to-work programs, today MDRC is also studying public school reforms, employment 
programs for ex-offenders and people with disabilities, and programs to help low-income 
students succeed in college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Child Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the 
United Kingdom, MDRC conducts its projects in partnership with national, state, and 
local governments, public school systems, community organizations, and numerous pri- 
vate philanthropies. 




